Thanks to Tim Hall, lots of people from Oracle community, shared their favourite Oracle feature as blog posts. I’ve decided to dedicate my OTN Appreciation Day blog post to Oracle Wait Interface. I know that the topic was not limited to Oracle database, and I could share something about Oracle Enterprise Manager (my main focus area), but Oracle database is the entry point for me and it’s the reason why I started to use other Oracle products (including Oracle Enterprise Manager).

Most of the time, people ask me why I prefer to work with Oracle database. It has great features like having reliable infrastructure, flashback, partitioning, in-memory column store etc. It comes with a great backup and recovery tool (RMAN) and it supports PL/SQL (great way to deal with data). All of these features (and the ones I didn’t count here) helps Oracle database to be the best relational database management system, but when I think about Oracle, the performance monitoring features are my favourite, and Oracle Wait Interface is the foundation stone of those features.

Each time a process waits for something, Oracle collects statistics about the wait. Oracle Wait Interface provides statistics for these waits, and these statistics can be accessed trough a collection of dynamic performance views or an extended SQL trace file. This information is invaluable to monitor sessions and guides you to find the root cause of many performance problems.

For more information about Oracle Wait Interface, you may read: “Oracle Wait Interface: A Practical Guide to Performance Diagnostics & Tuning” book.

Enterprise Manager Cloud Control provides a centralized job scheduling system and it’s possible to control it through both web interface and EMCLI. On the other hand, it seems web interface provides more control over the past jobs, for example it’s not possible to delete a job run with EMCLI, while it’s possible to do it on web interface:

EMCLI verb “get_jobs” reports all runs of the jobs, and “delete job” can delete job and its all runs but it’s not possible to delete a specific run of a job. For example, if you want to delete hundreds of “failed” (or “skipped”) runs of a job, EMCLI won’t help you and doing it through the web interface will be very time consuming.

So what is the solution? We need to make our hands dirty and call EM_JOB_OPS.delete_job_run procedure directly on the repository database.

Here’s a simple PL/SQL script to clean up failed jobs of a user:

BEGIN
  FOR c IN
  (SELECT scheduled_time
  FROM MGMT$JOB_EXECUTION_HISTORY
  WHERE job_owner=:1
  AND job_id     =:2
  AND status     ='Failed')
  LOOP
    EM_JOB_OPS.delete_job_run( :2, c.scheduled_time );
  END LOOP;
  COMMIT;
END;

Login to the Enterprise Manager repository database as SYSMAN, enter “job_owner” and “job_id” as bind variables. It will delete all failed runs of the job. You can query MGMT$JOB_EXECUTION_HISTORY view or use “emcli get_jobs” to learn the job owner and job ID information.

When you need to monitor web services, you can use EM beacons. A beacon is a target that allows the EM Agent to remotely monitor services. You can add beacons to any EM agent using “Add Non-Host Targets by Specifying Target Monitoring Properties” option in the Add Targets Manually page. After adding a beacon, you can configure it to fetch a web page and test the status of the service according to the result of the web transaction.

One of my blog reader said that they have a list of URLs and they need to know which beacon server is polling each URL by querying the URL. We need to query four views to get this information: mgmt_targets, mgmt_bcn_target, mgmt_bcn_step_defn, mgmt_bcn_step_props.

You can see the relation between these views in the diagram, and here’s the list of URLs monitored by all beacons:

SELECT t.HOST_NAME,
  t.TARGET_NAME,
  bsp.STRING_VALUE,
  bsp.NAME
FROM MGMT_TARGETS t,
  MGMT_BCN_TARGET bt,
  MGMT_BCN_STEP_DEFN bsd,
  MGMT_BCN_STEP_PROPS bsp
WHERE bt.BEACON_TARGET_GUID = t.TARGET_GUID
AND bsp.STEP_GUID = bsd.STEP_GUID
AND bsd.TARGET_GUID = bt.TARGET_GUID
AND bsp.NAME = 'url';

I’ve started to test Oracle Database Cloud Services. I applied for trial, it’s accepted (you can also get a trial account, there’s no special requirement for it). After I got my welcome emails, I created a database service. As expected, it was very easy. All I needed is to click a couple of times, enter mandatory information and then wait until my Database is provisioned.

Oracle creates a VM for each database because they provide SSH access to the server. This gives you more power than standard PaaS but I don’t know what happens if you make some modification on the OS level such as upgrading packages, changing kernel settings etc…

Anyway, after I created a Database Service, I wanted to connect to EM but I see that I couldn’t access the page. First, I though it’s a bug, then I remembered our company firewall rules are very strict and it’s related with our firewall, then I noticed that almost all ports (expect SSH) are blocked.

So after you create the database service, click to menu located at the right side of the database service name, and select “Access Rules” to open the access rules page.

In this page, you can see the predefined rules and their status. The last two rules are internal and you shouldn’t edit/disable them (read the description), you can enable or disable the other rules.

Here are the rules and what they are used for:

• ora_p2_dblistener (port 1521), used by SQL*Net (as you know).
• ora_p2_http (port 80), should be used for HTTP connections but I don’t see any service using it.
• ora_p2_httpssl (port 443), used for HTTPS connections, including Oracle REST Data Services, APEX and Oracle DBaaS Monitor.
• ora_p2_httpadmin (port 4848), should be used for GlassFish administration server, I do not know why it’s predefined.
• ora_p2_dbconsole (port 1158), used by Enterprise Manager 11g Database Control. Enable it if your Database is 11g.
• ora_p2_dbexpress (port 5500), used by Enterprise Manager Database Express 12c. Enable it if your Database is 12c.

You can also create your own rules. For example, you can create a rule to enable access for all ports from your own computer. Click on “create rule”, it will open a new window, give a name for the rule, enter the source IP (or just select public interface for all IPs), enter a port or a port range (for example 1-65535) and the protocol. As you can see, if I entered my own IP to give access to all ports on the database server.

So far, I liked what I see and I will probably write some more blog posts about Oracle Database Cloud Service in the following days

I noticed an interesting security problem (exploit?) on Oracle Enterprise Manager Cloud Control 13cR2 (I tested on EM13cR1 and it also exists on there). When you create an Enterprise Manager administrator, you need to grant some special privileges to that administrator if you want them to access the performance pages, but it seems there’s an alternative way to access the performance pages without requiring extra privileges.

Let’s say I created a new administrator with default roles (EM_USER and PUBLIC), and granted “Connect Target Read-only” for a target “RAC database”. I also granted access to a named credential or shared database login credentials.

When the new administrator goes to “RAC database home”, they will see that all links related with performance pages are greyed out (disabled) on the menu. It’s the expected situation. We do not want them to access the performance pages and it seems they can’t.

On the other, when the new administrator, goes to database targets page, and click “the database” on the database load map, they will directly land on the performance page!

All menus related with the performance page is still disabled but our new administrator can just click on “Top Activity” link and access the top activity page too!

They can even see the running SQLs while they hold the pointer over the SQL ids on the Top SQL list. On the other hand, they won’t be able to connect nodes individually. If the target was not a RAC database, they will be able to land the performance page of the instance. So the problem is “database load map” does not check the privileges of the administrator.

I couldn’t decide if this is a security exploit or maybe a hidden feature

After I have installed the EM13c R2, I noticed that I couldn’t access the SQL Monitoring page. I tested the SQL monitoring page for 11g to 12c database targets, and it gave the same error: ORA-06553: PLS-306: wrong number or types of arguments in call to ‘REPORT_SQL_MONITOR_LIST’. As you can guess I hit a bug, good thing is someone else already raised a service request and Oracle released Patch 24914115 to fix the bug. When I examined the patch, I saw that it’s also released for DB Plugin (OMS) 12.1.0.8.161031 Bundle. So if you applied this 161031 bundle for EM13 R1, you probably have the same bug, and you need to apply the patch.

I was planning to say a few words about the poor testing of EM13c releases, but I’m sure everyone already knows it, so I’ll just tell how I applied the Patch 24914115.

First of all, we need to update OPatch (in the Middleware Home) and OMSPatcher. To update OPatch, go to the Patch 6880880, and pick 13.9 release. It’s the OPatch for Enterprise Manager. I don’t think that Oracle Database will have 13.9 release so it seems 13.9 is reserved for Enterprise Manager 13c.

OPatch 13.9 is a different than previous OPatch releases. It comes with an installer. Download “p6880880_139000_Generic.zip”, unzip it to a temporary directory. You’ll see a file named “opatch_generic.jar”. Using JDK of Oracle Middleware Home, run the jar file:

unzip p6880880_139000_Generic.zip -d /tmp
cd /tmp
$OMS_HOME/oracle_common/jdk/bin/java -jar opatch_generic.jar -silent oracle_home=/u02/oracle/Middleware

According to the readme file, the jar file requires two parameters. First one is “-slient” (I didn’t tested what happens if I don’t give -slient parameter), and second one is oracle_home. The oracle_home parameter should point to the Oracle Middleware Home.

To update OMSPatcher, go to the Patch 19999993, and pick the correct Enterprise Manager release. Download the patch file, rename the old OMSPatcher folder and unzip the new OMSPatcher:

cd $OMS_HOME
mv OMSPatcher OMSPatcher.bak
unzip p19999993_132000_Generic.zip -d $OMS_HOME

After you update them, check the releases of the OMSPatcher and OPatch:

$OMS_HOME/OMSPatcher/omspatcher version
OMSPatcher Version: 13.8.0.0.1
OPlan Version: 12.2.0.1.6
OsysModel build: Tue Nov 08 17:52:30 PST 2016

OMSPatcher succeeded.

$OMS_HOME/OPatch/opatch version
OPatch Version: 13.9.1.0.0

OPatch succeeded.

So we upgraded both the OMSPatcher and the OPatch. Now we can download the patch and apply it to our system.

Go to the Patch 24914115, download the patch for EM13c R2, unzip the file for a temporary folder, switch that folder, set ORACLE_HOME variable to your Oracle Middleware Home, and run the omspatcher. I’ll run it with analyze parameter to be sure that the patch has no conflicts and all prerequisites are already met:

unzip p24914115_132100_Generic.zip -d /home/oracle/patches
cd /home/oracle/patches/24914115
export ORACLE_HOME=/u02/oracle/Middleware
$ORACLE_HOME/OMSPatcher/omspatcher apply -analyze
OMSPatcher Automation Tool
Copyright (c) 2016, Oracle Corporation.  All rights reserved.


OMSPatcher version : 13.8.0.0.1
OUI version        : 13.8.0.0.0
Running from       : /u02/oracle/Middleware
Log file location  : /u02/oracle/Middleware/cfgtoollogs/omspatcher/...log

OMSPatcher log file: /u02/oracle/Middleware/cfgtoollogs/omspatcher/24914115/...log

Please enter OMS weblogic admin server URL(t3s://...com:7102):> 
Please enter OMS weblogic admin server username(weblogic):> 
Please enter OMS weblogic admin server password:>

Enter the weblogic admin server, username and password. OMSPatcher will detect the admin server and username, so you probably just accept them with pressing enter. All you need is to enter the admin server password.

Prerequisites analysis summary:
-------------------------------

The following sub-patch(es) are applicable:

             Featureset   Sub-patches                                     Log file
             ----------   -----------                                     --------
  oracle.sysman.top.oms      24914115                        24914115_opatch...log

If you see a message similar to above, you’re ready to apply the patch. Issue “omspatcher apply” command (without -analzye parameter) to apply the patch. After applying the patch, you need to restart OMS to make changes take effect. That’s all, so we can access the SQL Monitoring page!

While playing with Oracle 12.2 Database on the Oracle Cloud, I noticed that SQL*Plus has a new feature which would be very useful for all DBAs. Now SQL*Plus can keep history of the commands executed, and let you see and run these commands from the history. Its usage and functionality is very similar to Unix history tool, but do not think that pressing the UP arrow will return to previous commands

The history feature is not enabled by default, so after you start SQL*Plus, you need to enable history by running “set hist[ory] on”:

SET HIST ON

After you issue the above command, all commands (except history commands) will be stored in the history list. The default length of the history list is 100. It means it stores 100 entries, and then first entry will be deleted to make space for the 101st one. You can write a number instead of “on” keyword, so the number will be the size of the history. For example, “set hist 10” command will enable history which will keep only last 10 entries.

You can run “show hist” to be sure that the history is enabled:

SHOW HIST
history is ON and set to "100"

You can list all entries in the history list by running “hist[ory]”, and run any command by its number:

HIST
  1  show hist
  2  select * from v$version;
  3  select * from v$datafile;
  4  desc v$datafile

HIST 1 RUN
history is ON and set to "10"

You can delete all history by running “hist clear”:

HIST CLEAR
HIST
SP2-1651: History list is empty.

As I see, it keeps the commands in memory, it doesn’t store it in a file like Unix shells do. So after you exit from SQL*Plus, history is cleared.

For more information, you can check oracle documentation

In the official backup and recovery documents of Oracle Enterprise Manager Cloud Control, when omsca tool is used for recovery, it is used with “-as” and “-ms” parameters but they are not well documented. For example:

Some parameters have no need any explanation but what are “-as” and “-ms”? Can we use them separately?

-AS means we want to recover the “OMS weblogic admin server”. The admin server operates as the central control entity for the configuration of the entire Enterprise Manager domain.

-MS means we want to recover the managed server hosting Oracle Management Service (OMS). It’s a mandatory parameter for recovery operations!

It’s possible to recover managed servers without recovering admin server of EM13c. For example, if you have Multi OMS site and lost one of your additional OMS, you can recover it with “-ms” parameter.

If you need to recover admin server, you also need to recover the managed server on the host, so you should use “-as” and “-ms” parameters together.

When creating a Database Cloud Service, you have an option to select “cloud storage” as a backup destination. In this case, the database backups will be stored in “storage containers”. As I see, lots of Oracle Cloud users have problem with setting up storage containers as backup destination.

It’s possible to create a storage container before you create the database service or you can create it while provisioning the DB service. Whichever method you chose, you should first set up the Replication Policy.

Go to the home page of “Oracle Storage Cloud Service”, on the up right side, you’ll see the action menu button (). Click on it and set the replication policy. Be careful about choosing the data centers and replication policy, because the replication policy can not be changed once it is set!

In the same page, you can see a grayed out text box containing REST Endpoint. Copy this information (for me, it’s https://bilyoner.storage.oraclecloud.com/v1/Storage-bilyoner). We’ll use it for creating storage containers.

So now we can create a database cloud service which will backup to storage containers. On the details page, we need to enter Cloud Storage Container name. We will add the name of our new container to the REST point of our Oracle Storage Cloud Service. For example:

https://bilyoner.storage.oraclecloud.com/v1/Storage-bilyoner/test11storage

We’ll also need to mark checked the Create Cloud Storage Container option. Then we can click next, review our responses and click create button to create the service.

If you want to create storage containers manually, you can use the Oracle Cloud Storage console. Its URL is storageconsole.DATACENTERCODE.oraclecloud.com. My cloud storage center is EM2, so the URL of my cloud storage console is https://storageconsole.em2.oraclecloud.com. To be able to login, we need to enter the REST point of our Oracle Storage Cloud Service, and our cloud account credentials (as cloud storage username and password).

It has a very simple interface, just let you see the contents of the containers, delete unused containers and create new ones. In my next post, I’m planning to demonstrate how we can use REST calls to create storage containers.

I was waiting the latest PSU for Oracle Database 11.2.0.4 on the Oracle Database Cloud Service, and today I noticed that it’s available. So let’s see how we can update our Oracle Databases in the cloud.

First we go to the database home to see if any updates is available. As you can see, in the administration box, there is an available patch. We click on it to see the details of the patch.

It’s the PSU Update 11.2.0.4.161018! I can read the readme file of the patch (from Oracle support) when I click the readme link. On the right side, there’s an action menu to run a precheck (prerequisites check) and to apply the patch. I run the precheck first, so I can be sure that I won’t have any problem when applying the patch.

Oh, the prerequisites checks failed!

Honestly, this is not my first time to apply a patch on Oracle Database Cloud Service. In my previous try about 2 or 3 months ago, I applied the latest PSU without any problem. It was a very smooth process, took less than hour. This time, the prerequisites failed.

Unfortunately, database home page doesn’t show any details about which prerequisite is failed. I wanted to try my chance with command line tools. Oracle Cloud provides dbaascli tool to manage Database Service including patching but some of its functions require root privileges. So I logged in to my cloud server via SSH as “opc” user – this user has sudo rights so I can switch to root user. And I issued “dbaascli dbpatchm –run -prereq” command to precheck the avaliable database patch.

[root@TEST11 ~]# dbaascli dbpatchm --run -prereq
DBAAS CLI version 1.0.0
Executing command dbpatchm --run -prereq -cli
/var/opt/oracle/patch/dbpatchm -prereq -cli
Starting DBPATCHM
Logfile is /var/opt/oracle/log/dbpatchm/dbpatchm_2016-12-30_05:34:12.log
Config file is /var/opt/oracle/patch/dbpatchm.cfg

INFO: cdb is set to : no 
INFO: dbversion detected : 11204 
INFO: patching type : psu 

WARN: inventory check: patch 23698582 installed is a oneoff - which could be lost in upgrade process 
WARN: and it can be reinstalled manually if needed, post upgrade. 
WARN: inventory check: patch 19272701 installed is a oneoff - which could be lost in upgrade process 
WARN: and it can be reinstalled manually if needed, post upgrade. 
WARN: inventory check: patch 20475845 installed is a oneoff - which could be lost in upgrade process 
WARN: and it can be reinstalled manually if needed, post upgrade. 
WARN: inventory check: patch 19469538 installed is a oneoff - which could be lost in upgrade process 
WARN: and it can be reinstalled manually if needed, post upgrade. 
WARN: inventory check: patch 22366322 installed is a oneoff - which could be lost in upgrade process 
WARN: and it can be reinstalled manually if needed, post upgrade. 
WARN: inventory check: patch 19490948 installed is a oneoff - which could be lost in upgrade process 
WARN: and it can be reinstalled manually if needed, post upgrade. 
WARN: inventory check: patch 20869721 installed is a oneoff - which could be lost in upgrade process 
WARN: and it can be reinstalled manually if needed, post upgrade. 
WARN: inventory check: patch 18043064 installed is a oneoff - which could be lost in upgrade process 
WARN: and it can be reinstalled manually if needed, post upgrade. 
WARN: inventory check: patch 23745609 installed is a oneoff - which could be lost in upgrade process 
WARN: and it can be reinstalled manually if needed, post upgrade. 
WARN: inventory check: patch 23177551 installed is a oneoff - which could be lost in upgrade process 
WARN: and it can be reinstalled manually if needed, post upgrade. 
WARN: inventory check: patch 23054359 installed is a psu - upgrade will bring in newer s/w bits 
INFO: there are interim patches installed - please check /var/opt/oracle/log/dbpatchm/invlog for details 
oss_patch: Database version is required 
dbpatchm Execution completed

No luck! I found a thread on Oracle Community saying we need to update dbaascli package (cloud tooling).

So I tried to check my version:

[root@TEST11 ~]# dbaascli dbpatchm --run -list_tools
DBAAS CLI version 1.0.0
Executing command dbpatchm --run -list_tools -cli
/var/opt/oracle/patch/dbpatchm -list_tools -cli
Starting DBPATCHM
Logfile is /var/opt/oracle/log/dbpatchm/dbpatchm_2016-12-30_05:40:51.log
Config file is /var/opt/oracle/patch/dbpatchm.cfg

INFO: cdb is set to : no 
INFO: dbversion detected : 11204 
INFO: patching type : psu 

dbpatchm Execution completed

Not very informative at the beginning! Then I read the log file and see that there’s an update available for dbaascli:

2016-12-30 05:40:53 - Output from cmd rpm -qa|grep -i dbaastools run on localhost  is:
2016-12-30 05:40:54 - dbaastools-1.0-1+16.4.5.0.0_161107.1102.x86_64
2016-12-30 05:40:54 - cmd took 1 seconds
2016-12-30 05:40:54 - INFO: rpmstr is : dbaastools-1.0-1+16.4.5.0.0_161107.1102.x86_64 
2016-12-30 05:40:54 - INFO: rpmdate is : 161107 
2016-12-30 05:40:54 - INFO: rpmdate returned from get_rpm_date : 161107 
2016-12-30 05:40:55 - INFO: total rpm's available for this version: 2 
2016-12-30 05:40:55 -    --------------
2016-12-30 05:40:55 -    | rpm      : 16.4.3.1_161117.0929
2016-12-30 05:40:55 -    | dated    : 161117
2016-12-30 05:40:55 -    | description: DBAAS Tools for Oracle Public Cloud
2016-12-30 05:40:55 -    --------------
2016-12-30 05:40:55 -    --------------
2016-12-30 05:40:55 -    | rpm      : 16.4.5.1_161213.1526
2016-12-30 05:40:55 -    | dated    : 161213
2016-12-30 05:40:55 -    | description: DBAAS Tools for Oracle Public Cloud
2016-12-30 05:40:55 -    --------------

According to the Oracle documents, I need to update the dbaascli package using dbaascli itself:

[root@TEST11 ~]# dbaascli dbpatchm --run -toolsinst -rpmversion=16.4.5.1_161213.1526
DBAAS CLI version 1.0.0
Executing command dbpatchm --run -toolsinst -rpmversion=16.4.5.1_161213.1526 -cli
/var/opt/oracle/patch/dbpatchm -toolsinst -rpmversion=16.4.5.1_161213.1526 -cli
Starting DBPATCHM
Logfile is /var/opt/oracle/log/dbpatchm/dbpatchm_2016-12-30_05:47:10.log

WARN : Parameter toolsinst passed via cmdline is not a valid parameter. Please check the usage.
Config file is /var/opt/oracle/patch/dbpatchm.cfg

INFO: cdb is set to : no 
INFO: dbversion detected : 11204 
INFO: patching type : psu 

INFO: existing dbaastools version - dbaastools-1.0-1+16.4.5.0.0_161107.1102.x86_64
 
INFO: updated dbaastools rpm to - dbaastools.rpm 
dbpatchm Execution completed

There’s a warning but it seems it’s updated. I execued “dbaascli dbpatchm –run -list_tools” again, check the logs and saw that it’s updated.

I used dbaascli to check prerequisites and see the same output with my previous one. So I felt like it didn’t work. To be sure, I also tried to check prerequisites on the Oracle Database Cloud Service Console, and it completed OK this time.

I tried to compare logs of my previous dbaascli runs, and I noticed that even you use web console to patch your database or check prerequisites of a patch, and I discovered that my first attempt failed because of dbaastools version!

2016-12-29 06:59:45 - INFO: Conflicting Patchid : 23745609 
2016-12-29 06:59:45 - INFO: Patchid 23745609 is NOT PSU 
2016-12-29 06:59:45 - INFO: Conflicting Patchid : 24006111 
2016-12-29 06:59:45 - INFO: Patchid 24006111 is PSU 
2016-12-29 06:59:45 - INFO: Conflicting patchid 23745609 to be checked with rollback string
2016-12-29 06:59:45 - INFO: Patchid 23745609 is PRESENT in rollback string 
2016-12-29 06:59:45 - force & ignore_patch_conflict are : 0 & 0
2016-12-29 06:59:45 - 
INFO: dbaastools version check
2016-12-29 06:59:45 - Current dbaastools version - dbaastools-1.0-1+16.4.5.0.0_161107.1102.x86_64

2016-12-29 06:59:45 - Output from cmd rm -f /var/opt/oracle/log/dbpatchm/inst_type.txt run on localhost  is:
2016-12-29 06:59:45 - cmd took 0 seconds
2016-12-29 06:59:46 - INFO: DB status is: 1 and is_stdby is set to : no
2016-12-29 06:59:47 - INFO: dbversion is 11.2.0.4.160719 
2016-12-29 06:59:48 - 
INFO: recommended version: 161117, required version: 161117 and url: https://storage.us2.oraclecloud.com/v1/dbcsswlibp-usoracle29538/dbaas_patch/dbaastools.rpm 
2016-12-29 06:59:48 - ERROR: please update the dbaastools version from https://storage.us2.oraclecloud.com/v1/dbcsswlibp-usoracle29538/dbaas_patch/dbaastools.rp
m
2016-12-29 06:59:48 - 
INFO: cstate updated for 1

At least I found where the logs are located!

I clicked “patch” on the action button. It asks me to enter a note and if I want to force the patch apply, I just entered “it’s so easy to patch :)” and click “patch” button.

I waited about 25 minutes to see the latest patch is applied and database is up! After all this work, I remembered that some people say there’s no need for DBAs for managing databases in the cloud

On Oracle Community forum, I’ve seen a good question about using EMCLI to add targets. The forum user says that they decided to name the database targets with combining db_name and hostname. As you may know, when you add a database target, EM also creates a database system for the new target (or assign it with an existing one). The new database system’s name is generated by adding “_sys” to the database target name. Let’s we add a database target named TEST, EM will create a database system as TEST_sys. If we name our database target as “TEST_test.gokhanatil.com”, EM will create a database system named “TEST_test.gokhanatil.com_sys”.

In my personal opinion, Enterprise Manager provides enough flexibility to report these information so I wouldn’t use this kind of naming system but that’s not the point. As you see, it works well for stand alone databases but when you add a standby database to this system, it becomes confusing. For example, your TESTSTBY_stby.gokhanatil.com will be part of TEST_test.gokhanatil.com_sys. So the forum user asks if we can give a specific name to the database system instead of default naming. For example, our database system will be named as TEST_sys although its members are TEST_test.gokhanatil.com and TESTSTBY_stby.gokhanatil.com.

If you use the web console, it’s possible to set a name for the database system, but as I see, there’s no option or parameter for it on emcli. So my solution is using a “hidden gem” in emcli verbs: rename_target.

Let’s say we add a target databases like this:

./emcli add_target -name="TEST_test.gokhanatil.com" \
-type="oracle_database" -host="test.gokhanatil.com" \
-prop="SID:TESTERE;MachineName:test.gokhanatil.com" \
-prop="OracleHome:/u01/OracleHomes/db/product/dbhome_1;Port:1521" \
-credentials="UserName:dbsnmp;password:welcome1;Role:normal"

Wait for a while and check if the database system is created:

./emcli get_targets -targets="TEST_test.gokhanatil.com%:%"
Status  Status           Target Type           Target Name
ID                                                                         
1       Up               oracle_database       TEST_test.gokhanatil.com   
1       Up               oracle_dbsys          TEST_test.gokhanatil.com_sys

Now let’s rename the new database system and check the result:

./emcli rename_target -target_name="TEST_test.gokhanatil.com_sys" \
-target_type="oracle_dbsys" -new_target_name="TEST_sys"

Target TEST_test.gokhanatil.com_sys successfully renamed to TEST_sys.

./emcli get_targets -targets="TEST%:%"
Status  Status           Target Type           Target Name
ID                                                                         
1       Up               oracle_database       TEST_test.gokhanatil.com   
1       Up               oracle_dbsys          TEST_sys

When we add the standby database of TEST_test.gokhanatil.com, Enterprise Manager will assign it to TEST_sys system. I hope this method solves the problem. We couldn’t give a specific name to the database system when adding the database target because there’s no (documented) parameter for it. So as a workaround, we renamed it after it’s created.

I prefer to write blog posts about the interesting questions on OTN. This blog post is one of them. There are usually more than one EM admins managing the systems, and you may want to track other users’ activity. Enterprise Manager Cloud Control provides auditing mechanism called “comprehensive auditing”. It’s not enabled by default for all actions because it may consume a lot of disk space.

If you want to enable it for all actions, you should use “emcli” tool:

./emcli login -username=SYSMAN  
./emcli enable_audit

After you enable comprehensive auditing for all actions, you can go to “setup” >> “security” >> “audit data” to see all audited actions.

The audit data page, provides filtering on audit records so I can easily list who deleted a target from the system.

If you haven’t enabled comprehensive auditing for all actions on Enterprise Manager, auditing is enabled only for login/logouts and infrastructure operations (such as removing EM key from repository, applying an update, creating CA etc..).

What if you haven’t enabled comprehensive auditing and someone decommission/remove an agent from the system? In this case, you can still find who did it (at least narrow the possibilities) by searching the access logs of OHS (Oracle Httpd Server installed as a part of Weblogic and EM13c).

The access logs are located in EM_INSTANCE_BASE/user_projects/domains/GCDomain/servers/ohs1/logs/ folder. You can check my blog post about log locations of EM13c.

You may wonder which keywords you’ll search. If you want to find the agent decommission, try to do it on EM13c, check the URL of the page, you’ll see something like “/em/faces/agentDecommision?target=….”. The agentDecommision is the keyword we’re looking for.

When we run “grep agentDecommision access_log”, we’ll see an output similar to the below text:

grep agentDecommision access_log  
access_log:192.168.16.225 - - [24/Jan/2017:23:29:28 +0300] "POST /em/faces/agentDecommision?target=xxxxx.com%3A3872&type=oracle_emd  
HTTP/1.1" 200 78 [ecid: 1.005Hhnt2eRP9HfGayxzW6G0001BT001tIp;kXjE] [User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3)  
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36]

We can easily say that the agent is decommissioned at 24/Jan/2017:23:29:28, by a Mac user whose IP is 192.168.16.225. Now we can search for logins on audit data of EM (using the audit data page) and identify the EM user who took the action.

You installed Oracle Enterprise Manager Cloud Control, added your production databases to the Enterprise Manager system. You set up the metric thresholds, monitoring templates and notification rules. Everything was looking OK until you got a warning email about “Current Open Cursors Count”. It was surprising because you didn’t set any threshold for it.

You checked the “Metric and Collection Settings” page to be sure that you didn’t set up any threshold, and you were right!

You wanted to edit the metric from the event details page, and all you got is “The metric data can not be found.” error. So what is happening?

This is a general question about that unreasonable “Current Open Cursors Count” warning event which is triggered when the open cursors count is greater than 1200 (default threshold value). The point is it is not generated by the metric thresholds of Enterprise Manager. This event is generated on the target database server!

If you’re in similar situation, connect to the target database, and check DBA_THRESHOLDS view. You’ll find the “Current Open Cursors Count” and other possible metrics and thresholds defined at the database server.

If you want to remove these thresholds, you need to DBMS_SERVER_ALERT.SET_THRESHOLD procedure. To clean up the threshold for “Current Open Cursors Count”, connect to the target database as SYS, and issue the following PL/SQL command:

begin   
DBMS_SERVER_ALERT.SET_THRESHOLD( metrics_id => 2104,   
warning_operator => NULL,warning_value => NULL,  
critical_operator => NULL,critical_value => NULL,  
observation_period => 1,consecutive_occurrences => 1,   
instance_name => 'YOUR_INSTANCE_NAME',   
object_type => DBMS_SERVER_ALERT.OBJECT_TYPE_SYSTEM,   
object_name => NULL );  
end;  
/

You can find the instance name from the DBA_THRESHOLDS. As you can see set_threshold procedure required the metric id as the first parameter. The metric id of “Current Open Cursors Count” is 2104, but what if you want to clean threshold of another database generated events such as “Average Users Waiting Counts”, “Logons Per Sec”? How will you find the IDs of these metrics?

To learn the metric IDs, we need to get the DML of the DBA_THRESHOLDS view. You can use Oracle SQL Developer’s find database object feature to get the DML. Then copy the SELECT part of the DML, and add a.metrics_id column to the query. It’s a UNION query, and there are combined two selects. So you need to add “a.metrics_id” to the both select queries. When you run the query, you can see the defined thresholds with their ID.

The notifications emails of a “planned” downtime is one of the annoying things of monitoring systems. You forget creating blackout and you start a maintenance work, and you get lots of notifications mails. Most of the time, the target which goes down, also affect other targets so it will multiple the number of unwanted notifications mails. Good thing is, it’s very easy to create blackouts on EM13c. We can do it through web console, emctl or emcli tools. A one-liner is enough for it:

emctl start blackout BlackOutName -nodeLevel

The above code will create a blackout for the all targets on the server. We can achieve the same thing by EMCLI:

emcli create_blackout -name="BlackOutName" -reason="you know we need to do it" -add_targets="myserver:host" -propagate_targets -schedule="duration:-1" # indefinite

If we use emcli, we have more options such as creating repeating blackouts, entering a reason for blackout, enabling blackout for a group of targets (which resides on different hosts).

What if we need to create blackout for multiple targets. As I mentioned, EMCLI can be used to create blackout for groups. We can create groups on EM13c, and instead of passing names of all targets in a group, we can give the group name:

#
# Sample EMCLI Python script file to create blackout for multiple targets
#

# check number of arguments

if len(sys.argv) <> 2:
    print "Usage to start a blackout: emcli @multiblackout.py targets.csv blackout_name"
    print "Usage to stop a blackout: emcli @multiblackout.py stop blackout_name"
    exit()

blackout_name = sys.argv[1].upper()

# login to OMS

login( username="SYSMAN", password="yoursupersecretpassword" )

if sys.argv[0].lower() == "stop":
    stop_blackout( name= blackout_name )
    # comment below line to keep old blackouts
    delete_blackout( name = blackout_name )
    print "%s blackout stopped and deleted." % blackout_name
    exit()

# open file for reading

f = open( sys.argv[0], 'r' )

# variable to keep all targets

alltargets = ""

# loop for each line of the file

for target in f:
    # build alltargets string
    alltargets += target.replace("\n","") + ";"

create_blackout( name = blackout_name, add_targets=alltargets, reason=blackout_name, schedule="duration:-1" )

The script accepts two parameters. First parameter is the path of the file containing the targets, the second parameter is the name of the blackout. The targets file should be something like this:

DBNAME1:oracle_database  
DBNAME2:oracle_database  
MYHOST5:host

After you create a blackout, you can stop (and delete) the blackout by running it again, but this time you need to enter “stop” as the file name:

./emcli @multiblackout.py /home/oracle/mytargets.txt TESTBLACKOUT   
./emcli @multiblackout.py stop TESTBLACKOUT

If you have any questions about the script, please do not hesitate to ask. By the way I’m aware that the script has lack of error handling, can be written more efficent but I’m not trying to provide a script library to you

I’m sharing a simple version so you can write your own (and better) script.

You know that patch numbering has been changed since November 2015, and the new format replaces the numeric 5th digit of the bundle version with a release date in the form “YYMMDD”. Let’s say you applied PSU 160719 to your 11.2.0.4 database, the exact version of your database becomes 11.2.0.4.160799. We also know that PSUs do not change the Oracle release version information that can be queried from v$version (Doc ID 861152.1), so when you query your database, you still see 11.2.0.4.0:

select * from v$version;  
  
BANNER  
--------------------------------------------------------------------------------  
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production  
PL/SQL Release 11.2.0.4.0 - Production  
CORE 11.2.0.4.0 Production  
TNS for Linux: Version 11.2.0.4.0 - Production  
NLSRTL Version 11.2.0.4.0 - Production

On the other hand, when you list your database targets on Oracle Enterprise Manager, it shows the exact version of Oracle Database.

I wondered how EM13c gets the information, and saw that it uses different queries depending on version of the target database. So first it queries v$version and then if the database version is lower than 11.2.0.2, it runs the following query:

SELECT NVL(  
  (SELECT version FROM  
    (SELECT version || '.' || id version  
    FROM dba_registry_history  
    WHERE NAMESPACE = 'SERVER'  
    ORDER BY ACTION_TIME DESC) WHERE rownum = 1),  
  (SELECT version FROM v$instance)) version  
FROM dual;

If the database version is greater than or equal to 11.2.0.2, and lower than 12.1.0.2, then it runs the following query:

SELECT NVL(  
  (SELECT version FROM  
    (SELECT version || '.' || id version  
    FROM dba_registry_history  
    WHERE NAMESPACE = 'SERVER'  
    AND BUNDLE_SERIES = 'PSU'  
    ORDER BY ACTION_TIME DESC) WHERE rownum = 1),  
  (SELECT version FROM v$instance)) version  
FROM dual;

If the target database version is greater than or equal to 12.1.0.2, then it runs the following query:

SELECT NVL(  
  (SELECT version FROM  
    (SELECT version || '.' || bundle_id version  
    FROM dba_registry_sqlpatch  
    WHERE BUNDLE_SERIES = 'PSU'  
    ORDER BY ACTION_TIME DESC) WHERE rownum = 1),  
  (SELECT version FROM v$instance)) version  
FROM dual;

I found it interesting so I wanted to share it. One day, if you need to get the exact version of your database (including PSU), it may be worth your while.

Here’s a quick and simple script which can help you to update a target property value of all targets running on a specific host. The script will expect you to enter 3 parameters: host name, property name and property value. Then it will search for the targets running on the given host name, and set the property to the given value:

emcli @setproperty.py hostname "propertyname" propertyvalue

It’s better to enter the property name in double quotes, because property names can contain spaces (for example “Line of Business”).

The script first checks the number of parameters, then login to OMS (enter your credentials – if you enter username but do not enter password, EMCLI will ask you to enter password when you run the script), gets the targets running on the given host (using EMCLI list command), and then executes set_target_property_value for each target. You may notice I give a non-default separator because some target names can contain colon (:) sign.

#  
# Sample EMCLI Python script file for mass update a property value  
#  
  
# check number of arguments 

if len(sys.argv) <> 3: 
  print 'Usage: emcli @setproperty.py hostname "propertyname" propertyvalue' 
  exit()

# assign the arguments to variables (for readability)

hostname = sys.argv[0]
propertyname = sys.argv[1]
propertyvalue = sys.argv[2]

# login to the OMS

login( username="SYSMAN", password="welcome1" )

# find the targets running on a given host

targets = list( sql="select target_name  || '@' || target_type as target from MGMT$TARGET where host_name = '" + hostname + "' and target_name <> host_name" )

# set the target property for each target 

for t in targets.out()['data']:
  print "Setting property " + propertyname + " for " + t['TARGET'].split(':')[0]
  set_target_property_value( property_records= t['TARGET'] + "@" + propertyname + "@" + propertyvalue, subseparator="property_records=@" )

You probably know that there’s an official github repository storing Dockerfiles and samples to build Docker images for Oracle products and Open Source projects. Now Oracle takes one more step to support Docker. Oracle Brings Oracle’s Flagship Databases and Developer Tools to the Docker Store (you can read the official announcement). Oracle Linux was already on the Docker Store. Oracle Database 12.1.0.2 Enterprise Edition, Oracle client tools, Oracle Weblogic, Oracle Coherence, Java 8 SE are also released on the Docker Store. Now you don’t need to build the image, you can download and use the image from the Docker Store.

To be able to use them, first you need to login to the Docker Store, select the Docker Image you want and click “get content” button. Then the Docker store will ask you enter your contact information (name, phone, company etc), and you have to accept “the Oracle license agreement”.

When we complete the form, we see the instructions (and a sample environment file) to use it. The environment file sets the required parameters such as database SID, password. So now we’re ready to get the image.

I login to the docker (using command line) and pull the image:

gatil:~ gokhan.atil$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: gokhanatil
Password: 
Login Succeeded
gatil:~ gokhan.atil$ docker pull store/oracle/database-enterprise:12.1.0.2
12.1.0.2: Pulling from store/oracle/database-enterprise
abf7e4428604: Pull complete 
00dfe33d8ca6: Pull complete 
bd2b2e675457: Pull complete 
599b3db5a3e4: Pull complete 
0e47b3ce0d30: Pull complete 
Digest: sha256:a5ce72142f33e667b330b2dba186f3554bcf88fd2f3c3853cf76873b08e56a89
Status: Downloaded newer image for store/oracle/database-enterprise:12.1.0.2

Then, I copied the sample environment file, modified the password, SID and domain, and run the docker image (my environment file name is testere.dat and my image name is testere):

docker run -d --env-file testere.dat -p 1521:1521 -p 5500:5500 -it --name testere --shm-size="4g" store/oracle/database-enterprise:12.1.0.2

Be careful that the shm-size must be 4g or higher.

One of the good thing is, we do not need to provide Oracle Database installation files for these images. Oracle put the all required files into the repository images. You may need to increase the memory resources of Docker. When I run the above command, I see that container created instantly but infact it requires to unzip the database software and create the database during on the first run. I waited for a while and followed the logs:

gatil:Desktop gokhan.atil$ docker logs -f testere
User check : root.
Setup Oracle Database
Oracle Database 12.1.0.2 Setup
Thu Apr 20 07:56:42 UTC 2017

Check parameters ......
log file is : /home/oracle/setup/log/paramChk.log
paramChk.sh is done at 0 sec

untar DB bits ......
log file is : /home/oracle/setup/log/untarDB.log
untarDB.sh is done at 130 sec

config DB ......
log file is : /home/oracle/setup/log/configDB.log
configDB.sh is done at 409 sec

Done ! The database is ready for use .
Thu Apr 20 07:56:42 UTC 2017
User check : root.
Setup Oracle Database

It’s done. I examined these logs and see that the scripts had some “ignorable” errors about patching. I also see that the database is a container database and it comes with a pluggable database named “PDB1”.

I connected my test database running on Docker, using my favorite SQL client (Oracle SQL Developer) and checked the status. Everything looks OK.

Over the last few years, Oracle has dedicated to cloud computing and they are in a very tough race with its competitors. In order to stand out in this race, Oracle provides more services day by day. One of the services Oracle offers to the end user is “Oracle Big Data Cloud Service – Compute Edition”. I examined this service by creating a trial account, and I decided to write a series of blog posts for those who would like to use this service.

In my opinion, the most difficult part of creating a Big Data ecosystem is to run many open source software projects together, and integrate them with each another. There are 3 major players on the market to help end-users to build an integrated and tested solution for big data: Cloudera, Hortonworks and MapR. Oracle has partnered with Cloudera to build the Oracle Big Data Appliance and Oracle Big Data Cloud Service. They also offer “Oracle Big Data Cloud Service – Compute Edition” based on Hortonworks. Creating “Oracle Big Data Cloud Service – Compute Edition” is simple. You get a ready-to-use big data cluster in about 15 minutes after giving the basic information such as the name of the cluster, the number of servers (nodes), CPU and disk sizes for each node, and the administrator password.

First, let’s create an “Oracle Big Data Cloud Service – Compute Edition”. After you create our test account for Oracle Cloud, you are log in to the “Oracle Cloud” dashboard. Using this dashboard you can see all your services and add new services at the same time.

When we click Create Service and select “Big Data – Compute Edition”, we land to “Oracle Big Data Cloud Service – Compute Edition” service page. On this page, we can click “create service” to launch a very simple wizard which has only 3 steps to create our big data cluster.

On the first step, we gave the name of the cluster and click next. I entered “bilyonveri” – it will be the name of my cluster.

On the second step, we enter the details of the service (number of nodes, CPU, etc.). We can use total 6 CPUs on a trial account, and since a node can have a minimum of 2 CPUs, we can create a cluster with a maximum of 3 nodes. Since the total disk size we can use is about 500 GB (I think it will be replaced with “read request”), you can give up to 80 GB of hdfs size per node. I recommend you to select the installation type “full”. If you select the “basic”, you get the most basic components which will only allow you to work with Spark.

Although we create a cloud storage container, our big data cluster does not use it.

On the last step, we confirm the settings and click “create” button.

When the service is created, you can see the overal status of your service by clicking on the service name. This page also allows you to stop and restart the service, set access permissions, and reach the Oracle “big data cluster console”.

The following services are installed on our cluster:

• HDFS (2.7.1.2.4)
• YARN + MapReduce2 (2.7.1.2.4)
• Tez (0.7.0.2.4)
• Hive (1.2.1.2.4)
• Pig (0.15.0.2.4)
• ZooKeeper (3.4.6.2.4)
• Spark (1.6.x.2.4)
• Zeppelin Notebook (0.6.0)
• Alluxio (1.2.0)
• BDCSCE Logstash Agent (0.0.1)
• Nginx Reverse Proxy (0.0.1)
• Spark Cloud Service UI (0.5.0)
• Spocs Fabric Service (0.1)

These services may be different for you (if you create it 1-2 months after I published this blog post). Big Data technologies (and the services related with big data) are changing very fast. Today, Oracle uses Hortonworks DP 2.4.2 – Current hortonworks data platform (2.6) and it comes with different services in the default setup. So these services will probably change when Oracle updates the version of HDP they use. In my next article, I will explain you what these services are used for.

To monitor our cluster, click on “big data cluster console”.

The console, which does not only show the general status of the system, also allows us to see the files in the HDFS file system, access the Zeppelin notebook (I will also write a separate blog post about it) and modify the settings of resource manager.

This is enough for this post, I will continue from where I left on my next post.

In my previous post, I gave a list of installed services on a “Oracle Big Data Cloud Service – Compute Edition” when you select “full” as deployment profile. In this post, I’ll explain these services and software.

HDFS: HDFS is a distributed, scalable, and portable file system written in Java for Hadoop. It stores data so it is the main component of the our cluster. A Hadoop (big data) cluster has nominally a single namenode plus a cluster of datanodes, but there are redundancy options available for the namenode due to its criticality. Both namenode and datanode services can run in same server (although it’s not recommended on a production environment). In our small cluster, we have 1 active namenode, 1 standby namenode and 3 datanodes – distributed to 3 servers.

YARN + MapReduce (v2): MapReduce is a programming model popularized by Google to process large datasets in a parallel and scalable way. is a framework for cluster resource management and job scheduling. YARN contains a Resource Manager and Node Managers (for redundancy we can create a standby Resource Manager). The Resource Manager tracks how many live nodes and resources are available on the cluster and coordinates which applications submitted by users should get these resources. Each datanode should have a nodemanager to run MapReduce jobs.

Hive: Hive is a data warehouse software facilitates reading, writing, and managing large datasets residing in HDFS. Hive gives an SQL-like interface to query data. These queries are implicitly converted into MapReduce or Tez jobs. It stores structured data like traditional databases. It requires a relational database to store the metadata for Hive tables. In our cluster, MySQL is used for hive metastore.

Tez: The Tez is the next generation Hadoop Query Processing framework written on top of YARN. It’s used by the Hive to increase performance.

Pig: Pig is a scripting platform for processing and analyzing large data sets. It has its own language (Pig Latin). Pig is ideal for Extract-transform-load (ETL), research on raw data, and iterative data processing.

ZooKeeper: ZooKeeper is an open source project that provides a centralized infrastructure and services that enable synchronization across a cluster. All other services in our cluster register themself to ZooKeper, so whenever a service is required, it can be found using Zookeper.

Spark: Apache Spark is a fast, in-memory data processing engine that allows data scientists to build and run fast and sophisticated applications on Hadoop. Spark Engine supports a set of high-level tools that support SQL-like queries, streaming data applications, complex analytics such as machine learning, and graph algorithms.

Zeppelin Notebook: Zeppelin Notebook is a web-based notebook that enables interactive data analytics. It can be used to make beautiful data-driven, interactive and collaborative documents with SQL, Spark, Scala and more. In our cluster, we have 3 zeppelin servers running on each node.

Alluxio: Alluxio (formerly Tachyon) is a memory-speed virtual distributed storage. It creates a storage layer between your applications and storage systems.

BDCSCE Logstash Agent: Logstash is a popular data collection engine with real-time pipelining capabilities. In our cluster, it is used to gather (and parse) logs of YARN Node Managers. The parsed logs are stored in HDFS. Normally, it’s not used on hadoop clusters.

Nginx Reverse Proxy:Nginx Reverse Proxy is used to provide authorization for Zeppelin Notebook. It’s also used for Big Data Console.

Spark Cloud Service UI: It’s written by Oracle to serve Zeppelin through Nginx proxy.

Spocs (or Spoccs) Fabric Service: It’s written by Oracle to provide Rest API for Jobs/Logs Management.

And of course, we have ambari service. Ambari is a completely open operational framework for provisioning, managing, and monitoring Hadoop clusters. It includes a set of operator tools and RESTful APIs that mask the complexity of Hadoop, simplifying the operation of clusters.

In my next post, I’ll talk about ambari console, and how we can add “missing views” to ambari.

This is my third blog post about Oracle Big Data Cloud Service – Compute Edition. I continue to guide you about the “Big Data Cloud Service – Compute Edition” and its components. In this blog post, I will introduce Ambari – the management service of our hadoop cluster.

The Apache Ambari simplifies provisioning, managing, and monitoring Apache Hadoop clusters. It’s the default management tool of Hortonworks Data Platform but it can be used independently from Hortonworks. After you create your big data service, SSH and 8080 (port used by Ambari) is blocked. You need to enable the rules to allow access through these ports. In my first blog post about Oracle Big Data Cloud Service – Compute Edition, I showed how to enable these ports.

Ambari has a client-server architecture. The Ambari agents run on each node in Hadoop cluster, and there’s an Ambari server to manage these agents and collect the data from them. In Oracle Big Data Cloud – Compute Edition, each node (we created 3 for trial) has ambari-server software but only the one ambari-server (which is on the first node) is configured and active. On the service overview page, you can see the IP address of the ambari-server.

The IP address of my first node is 141.144.22.114, so the URL of the Ambari server is https://141.144.22.114:8080. If we are already logged in to “Big Data Console”, the ambari page doesn’t ask any credentials but if we are not logged in to “Big Data Console”, we need to enter “bdcsce_admin” as username and the password of the admin user (we defined it while creating the service).

After we login to Ambari web interface, a detailed dashboard welcomes us. We can see performance graphs, alerts (warnings + criticals), existing services (on the left side) in this page.

We’ll use the top menu to access components of Ambari. On the very left, we see “ambari” logo and text, when we click it, we go to the current dashboard. On the right of ambari logo, we see our cluster name (bilyonveri) and “0 ops 2 alerts”. When we click “0 ops” text, we can see the operations running on background (such as starting/stopping services). There’s no operations running right now so we see “0”. When we click 2 alerts, we can see the details of current alerts.

On the top right side, we see link for dashboard, services (you can also same list on the left panel), hosts (to see the hosts in our cluster), alerts (to manage settings of alerts and see the current alerts), admin (to see software versions, service accounts, kerberos settings), views (matrix icon), and user menu. You can access “ambari management page” from the user menu by clicking “manage ambari”.

Each service has its own home page. You can access a service home page from the left panel or through the service menu on top. On that page, you can stop/start a service, turn on/off maintenance, move service components between nodes, run service checks, and delete the service (after we stop it). We can also edit configuration of the services from the “configs” link.

One of the first thing I noticed after I login to ambari web interface is, there are no views available. In normal conditions, there should be views for accessing Hive, Tez, Pig, Files and other services. I do not know the reason why Oracle didn’t include those views, maybe there are some compatibility issues but I want to show you how we can add one of them, the “Hive” view – so we can run Hive commands through Ambari web interface.

The Ambari views are java applications, and it’s easy to get their source codes from github, but it is a little bit complex to compile them for non-Java users (like me). So instead of dealing source codes, we’ll download ambari-server RPM from hortonworks, extract the jar file from the rpm, and then put that file in a special folder – Ambari continuously checks that special folder and load the view.

In my first post, I said Oracle uses Hortonworks DP 2.4.2. I found this information from admin menu, versions page. Now we know the version, we can go to hortonworks installation documents to get the URL of hortonworks repository for Oracle Linux 6. Please note that “Oracle Big Data Cloud Service – Compute Edition” runs on Oracle Linux 6.8.

We need to login to the first node (which hosts ambari-server) via SSH, switch to root user and download the ambari repository configuration, and use it to download installation RPM of ambari server:

sudo su -

wget -nv http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.4.2.0/ambari.repo -O /etc/yum.repos.d/ambari.repo

yum repolist

After the above command, yum will check the repositories and update the list of available packages. Because we do not want to touch current installation, we will use “yumdownloader” tool to download the RPM.

yumdownloader ambari-server --destdir=/root

After the above command, the “ambari-server-2.4.2.0-136.x86_64.rpm” file will be downloaded into the /root folder. I recommend to remove the ambari repository when it’s done, to prevent any confusion when applying “potential patch updates” to the system.

rm /etc/yum.repos.d/ambari.repo
yum repolist

We go to “/root” folder and extract the content of the RPM, then go the view folder:

rpm2cpio ambari-server-2.4.2.0-136.x86_64.rpm | cpio -idmv

cd var/lib/ambari-server/resources/views/
ls -lash
total 568M
4.0K drwxr-xr-x  2 root root 4.0K Jun 15 07:33 .
4.0K drwxr-xr-x 11 root root 4.0K Jun 15 07:33 ..
900K -rwxrwxrwx  1 root root 897K Nov 23  2016 ambari-admin-2.4.2.0.136.jar
 44M -rwxrwxrwx  1 root root  44M Nov 23  2016 capacity-scheduler-2.4.2.0.136.jar
 40M -rwxrwxrwx  1 root root  40M Nov 23  2016 files-2.4.2.0.136.jar
2.4M -rwxrwxrwx  1 root root 2.4M Nov 23  2016 hawq-view-2.4.2.0.136.jar
 95M -rwxrwxrwx  1 root root  95M Nov 23  2016 hive-2.4.2.0.136.jar
121M -rwxrwxrwx  1 root root 121M Nov 23  2016 hive-jdbc-2.4.2.0.136.jar
 34M -rwxrwxrwx  1 root root  34M Nov 23  2016 hueambarimigration-2.4.2.0.136.jar
 45M -rwxrwxrwx  1 root root  45M Nov 23  2016 pig-2.4.2.0.136.jar
 51M -rwxrwxrwx  1 root root  51M Nov 23  2016 slider-2.4.2.0.136.jar
1.2M -rwxrwxrwx  1 root root 1.2M Nov 23  2016 storm-view-2.4.2.0.136.jar
 48M -rwxrwxrwx  1 root root  48M Nov 23  2016 tez-view-2.4.2.0.136.jar
 47M -rwxrwxrwx  1 root root  47M Nov 23  2016 wfmanager-2.4.2.0.136.jar
 44M -rwxrwxrwx  1 root root  44M Nov 23  2016 zeppelin-view-2.4.2.0.136.jar

We will copy the hive-jdbc-2.4.2.0.136.jar file to “/var/lib/ambari-server/resources/views/” folder. Ambari will check the content of this folder automatically and load the jar file.

After we copy the file, “Hive View” will be automatically created and added to Ambari web interface. You may need to refresh the page to be able to see it. When we click “Hive View”, we’ll get an error “Service ‘userhome’ check failed: User: root is not allowed to impersonate bdcsce_admin”. Ambari runs as root user, we logged in as “bdcsce_admin” so ambari tries to access bdcsce_admin user home (HDFS) directory with root user. If you are a Linux user, you may say “root” can access everywhere but for HDFS, the king is “hdfs” user. So we have to configure hdfs and say that “root” can use any user (or bdcsce_admin).

We go the configs page of the HDFS, and search for “hadoop.proxyuser.root.users” parameter. We will see hadoop.proxyuser.root.users is oracle, we need to enter “*” (star) to enable root to access all users’ files. After we modify the setting, we click SAVE button.

You’ll see the HDFS services require restart to be able to apply configuration changes. We also need to restart other services depending on HDFS service. To restart all these services together, I click “restart all required” button on the actions menu on the left panel.

We wait until all services restarted and then try to click “Hive View” again. After the modification, we should be able to access Hive View with bdcsce_admin user. You can add other views but you may need to troubleshoot some errors. In my opinion, big data administrators should be familiar with all these troubleshooting process, so get used to it

In my next blog, I’ll mention about Zeppelin.