Monday, May 19, 2008

Creating a RAC database using standard Sql*plus

I'm writing this to avoid the hassle of using DBCA for creating a new RAC database. I started off by creating a non-RAC database (on one of the nodes in the RAC cluster, preferably node 1) using the conventional "create database" command. I then converted that to a RAC instance by setting RAC specific settings and finally registered the instance w/ the cluster.

Create a non-RAC database

Create a pfile
1) Create a pfile for a non-RAC database on node 1 (you may keep RAC specific parameters also in it since it will be later converted to a RAC database)
Here's a sample pfile (notice that I commented out the cluster_database and undo_tablespace parameters but left other RAC specific parameter on) --
PLINKTST2.__db_cache_size=4395630592
PLINKTST1.__db_cache_size=4395630592

PLINKTST2.__java_pool_size=16777216
PLINKTST1.__java_pool_size=16777216
PLINKTST2.__large_pool_size=16777216
PLINKTST1.__large_pool_size=16777216
PLINKTST2.__shared_pool_size=905969664
PLINKTST1.__shared_pool_size=905969664
PLINKTST2.__streams_pool_size=0
PLINKTST1.__streams_pool_size=0
##*.cluster_database=true
*.cluster_database_instances=2

*.compatible='10.2.0.4'

*.db_block_size=8192
*.db_create_online_log_dest_1='+PLINKTST_DATA'
*.db_domain=''

*.db_file_multiblock_read_count=32
*.db_name='PLINKTST'
*.db_unique_name='PLINKTST'

*.db_create_file_dest='+PWTEST_DATA'
*.db_recovery_file_dest='+PWTEST_DATA'
*.db_recovery_file_dest_size=53687091200
*.job_queue_processes=0
*.log_archive_dest_1='LOCATION=+PWTEST_DATA'

*.log_buffer=20971520
*.open_cursors=300
*.optimizer_mode='ALL_ROWS'
*.pga_aggregate_target=500M
*.processes=100
*.remote_login_passwordfile='exclusive'
*.resource_limit=TRUE
*.sga_max_size=1G

*.sga_target=1G
*.star_transformation_enabled='TRUE'
*.undo_management='AUTO'
*.undo_retention=3600

PLINKTST1.local_listener='LISTENER_PLINKTST1'
PLINKTST2.local_listener='LISTENER_PLINKTST2'
#PLINKTST1.undo_tablespace='UNDOTBS1'
#PLINKTST2.undo_tablespace='UNDOTBS2'

PLINKTST1.instance_number=1
PLINKTST2.instance_number=2
PLINKTST1.thread=1
PLINKTST2.thread=2

*.user_dump_dest='/d01/app/pwprd/product/10.2.0/pwprddb/admin/PLINKTST/udump' *.control_files='+PWTEST_DATA/plinktst/controlfile/plinktst.ctl' *.audit_file_dest='/d01/app/pwprd/product/10.2.0/pwprddb/admin/PLINKTST/adump' *.background_dump_dest='/d01/app/pwprd/product/10.2.0/pwprddb/admin/PLINKTST/bdump' *.core_dump_dest='/d01/app/pwprd/product/10.2.0/pwprddb/admin/PLINKTST/cdump' *.fal_server=PLINKTST
*.fal_client=PLINKTST
*.standby_file_management=AUTO


[Note: Use a node# in the instance name parameter (e.g. PLINKTST1) since it would later be converted into a RAC instance]


Startup nomount

PLINKTST1 > startup nomount

ORA-01078: failure in processing system parameters

ORA-00119: invalid specification for system parameter LOCAL_LISTENER
ORA-00132: syntax error or unresolved network name 'LISTENER_PLINKTST1'
PLINKTST1 >

Well, this needs a TNS entry for the local listener listed above (we can worry about the actual listener later - all it needs now is resolving the name LISTENER_PLINKTST1.

Open up the tnsnames.ora and add the following lines --
LISTENER_PLINKTST1 = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-rptdb01-pwprd)(PORT = 1522)) )

and start the instance.

PLINKTST1 > startup nomount

ORACLE instance started.
Total System Global Area 1073741824 bytes

Fixed Size 2144432 bytes
Variable Size 257247056 bytes
Database Buffers 792723456 bytes
Redo Buffers 21626880 bytes
PLINKTST1 >

Create the database

If we forgot to remove the "cluster_database" parameter earlier, we'll see an error while creating database as shown below.


PLINKTST1 > create database;
create database
*
ERROR at line 1:
ORA-01501: CREATE DATABASE failed
ORA-12720: operation requires database is in EXCLUSIVE mode

PLINKTST1 > show parameter cluster
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
cluster_database boolean TRUE
cluster_database_instances integer 2
cluster_interconnects string
PLINKTST1 >


Remove the parameter from the pfile; startup nomount and execute the create database again.
Once the database is created, execute all the usual good stuff - catalog, catproc, pupbld scripts (all under /rdbms/admin)



Converting into RAC instance

Add separate Undo Tablespaces

First, Create 2 UNDO tablespaces for the 2 instances.

PLINKTST1 > create undo tablespace UNDOTBS1 datafile '+PWTEST_DATA' size 100M autoextend on next 50M maxsize 500M;
Tablespace created.


PLINKTST1 > create undo tablespace UNDOTBS2 datafile '+PWTEST_DATA' size 100M autoextend on next 50M maxsize 500M;

Tablespace created.


Add new REDO thread and enable it

(otherwise it will give this error while starting the 2nd instance: ORA-01620: no public threads are available for mounting)


PLINKTST1 > alter database add logfile thread 2 group 10 '+PWTEST_DATA' size 100M;
Database altered.

PLINKTST1 > alter database add logfile thread 2 group 11 '+PWTEST_DATA' size 100M;
Database altered.


PLINKTST1 > alter database enable public thread 2;

Database altered.


RAC Specific parameters

Add all RAC specific parameters to the pfile (I simply uncommented the undo_tablespaces and the cluster_data parms as I already included those). If the RAC specific were not included earlier in the pfile, add those now. Here are the main ones -- *.cluster_database=true
*.cluster_database_instances=2

PLINKTST1.local_listener='LISTENER_PLINKTST1' PLINKTST2.local_listener='LISTENER_PLINKTST2'
PLINKTST1.undo_tablespace='UNDOTBS1'
PLINKTST2.undo_tablespace='UNDOTBS2'

PLINKTST1.instance_number=1
PLINKTST2.instance_number=2

PLINKTST1.thread=1
PLINKTST2.thread=2

Start the additional Listeners
We added a TNS entry for the LOCAL_LISTENER earlier. Lets start a new listener at that port (1522 in this case).
Update the following files::


listener.ora
----------------

LISTENER_PLINKTST_RPTDB01-PWPRD =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = vip-rptdb01-pwprd)(PORT = 1522)(IP = FIRST))
(ADDRESS = (PROTOCOL = TCP)(HOST = rptdb01-pwprd)(PORT = 1522)(IP = FIRST))
)
)

SID_LIST_LISTENER_PLINKTST_DB01-PWPRD =
(SID_LIST =
(SID_DESC =
(SID_NAME = PLSExtProc)
(ORACLE_HOME = /d01/app/pwprd/product/10.2.0/pwprddb)
(PROGRAM = extproc)
)

Start the above listener. Also, add the TNS entry for the new database.

tnsnames.ora
--------------------
PLINKTST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = vip-rptdb01-pwprd)(PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = vip-rptdb02-pwprd)(PORT = 1522))
(LOAD_BALANCE = yes)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = PLINKTST)
(FAILOVER_MODE =
(TYPE = SELECT)
(METHOD = BASIC)
(RETRIES = 180)
(DELAY = 5)
)
)
)

Shared pfile
Move the pfile to a shared location in ASM; restart instance 1; create all the appropriate directories on node2 and start the instance 2.
If you get the following error, make sure you've created the additional thread and enabled it (see the section "Add new REDO thread and enable it") --

PLINKTST2 > startup
ORACLE instance started.

Total System Global Area 1073741824 bytes
Fixed Size 2144432 bytes
Variable Size 257247056 bytes
Database Buffers 792723456 bytes
Redo Buffers 21626880 bytes
ORA-01620: no public threads are available for mounting
..
..
..
From node 1:
PLINKTST1 > alter database enable public thread 2;

Database altered.

..

The database should come up ok now.
PLINKTST2 > startup
ORACLE instance started.

Total System Global Area 1073741824 bytes
Fixed Size 2144432 bytes
Variable Size 257247056 bytes
Database Buffers 792723456 bytes
Redo Buffers 21626880 bytes
Database mounted.
Database opened.
PLINKTST2 >


This completes create a RAC database using SQL*Plus. Next we should register all the resources with CRS.


Registering with CRS
$ srvctl add database -d PLINKTST -o /d01/app/pwprd/product/10.2.0/pwprddb
$ srvctl add instance -d PLINKTST -i PLINKTST1 -n rptdb01-pwprd
$ srvctl add instance -d PLINKTST -i PLINKTST2 -n rptdb02-pwprd
$ srvctl add service -d PLINKTST -s PLINKTSTSVC -r "PLINKTST1,PLINKTST2"


Check for all the services just registered.
$ . oraenv
ORACLE_SID = [PLINKTST1] ? CRS

$ crs_stat |grep PLINK
NAME=ora.PLINKTST.PLINKTST1.inst
NAME=ora.PLINKTST.PLINKTST2.inst
NAME=ora.PLINKTST.PLINKTSTSVC.PLINKTST1.srv
NAME=ora.PLINKTST.PLINKTSTSVC.PLINKTST2.srv
NAME=ora.PLINKTST.PLINKTSTSVC.cs
NAME=ora.PLINKTST.db

All the new resources have been registered with CRS.

Tuesday, May 13, 2008

Moving SPFILE and CONTROLFILE from cooked FileSystem to ASM

You have a standby database (or any database for that matter) and it uses a pfile sitting under a non-standard directory (like your home directory). Ditto for controlfile. Now you plan to move the pfile to an spfile (under ASM) and the controlfile from your cooked filesystem to ASM. It seems rather easy, but I found something rather interesting while doing this exercise.

First, I used the standard syntax for creating the spfile --
PWPRDDR1 > create spfile='+PWPRD_DATA' from pfile;
File created.


All is well, and I could see a new spfile created under ASM.
ASMCMD [+PWPRD_DATA/PWPRDDR] > ls -lt PARAMETERFILE/
Type Redund Striped Time Sys Name
PARAMETERFILE UNPROT COARSE MAY 13 13:00:00 Y spfile.325.654606981


I also created an alias for this spfile.
ASMCMD [+PWPRD_DATA/PWPRDDR] > mkalias +PWPRD_DATA/PWPRDDR/PARAMETERFILE/spfile.326.65460698 spfilePWPRDDR.ora

And reference this alias in the init file under dbs directory.
echo "spfile='+PWPRD_DATA/PWPRDDR/spfilePWPRDDR.ora'" > $ORACLE_HOME/dbs/initPWPRDDR1.ora
(notice I'm doing this for a RAC standby)

Finally, I tried starting the database in nomount mode --
PWPRDDR1 > startup nomount
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+PWPRD_DATA/PWPRDDR/PARAMETERFILE/spfile.325.654601887'
ORA-17503: ksfdopn:2 Failed to open file +PWPRD_DATA/PWPRDDR/PARAMETERFILE/spfile.325.654601887
ORA-15056: additional error message
ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +PWPRD_DATA/pwprddr/parameterfile/spfile.325.654601887
ORA-17503: ksfdopn:2 Failed to open file +PWPRD_DATA/pwprddr/parameterfile/spfile.325.654601887
ORA-15012: ASM file '+PWPRD_DATA/pwprddr/parameterfile/spfile.325.654601887' does not exist
ORA-06512: at line 4



Oops! what went wrong? I spent some time redoing the steps but every time it resulted in the same error.
Well, the mistake I was doing was that the database was started using a pfile sitting under non-standard location (my work area folder). The syntax for spfile creation needs to know that.

The simple fix therefore was:
1) I restarted the database (mount mode).
PWPRDDR1 > startup nomount pfile=/home/pwprd/ravi/initPWPRDDR.ora

2) Created an spfile using the following syntax.
PWPRDDR1 > create spfile='+PWPRD_DATA' from pfile='/home/pwprd/ravi/initPWPRDDR.ora';
File created.


3) Removed and (re)created the alias in ASM (asmcmd) pointing to the new spfile.
ASMCMD [+PWPRD_DATA/PWPRDDR] > rmalias spfilePWPRDDR.ora
ASMCMD [+PWPRD_DATA/PWPRDDR] > mkalias +PWPRD_DATA/PWPRDDR/PARAMETERFILE/spfile.326.654614871 spfilePWPRDDR.ora


4) Finally started the instance (without using the pfile this time).
PWPRDDR1 > startup nomount


The next task is to move the controlfile into ASM. This was rather straight -- My current controlfile is sitting under /home/pwprd/ravi.
PWPRDDR1 > show parameter control_file
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_files string /home/pwprd/ravi/stby_PWPRD.ctl


From a different session, while the database (standby) is still mounted, I create a backup of the controlfile using RMAN.
RMAN> copy current controlfile to '+PWPRD_DATA/PWPRDDR/CONTROLFILE/control01.ctl';
Starting backup at 13-MAY-08
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
copying current control file
output filename=+PWPRD_DATA/pwprddr/controlfile/control01.ctl tag=TAG20080513T140358 recid=138 stamp=654617041
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
Finished backup at 13-MAY-08

[Note: You may also use DBMS_FILE_TRANSFER.COPY_FILE procedure to copy files. For doing that, you'd need to create 2 directory objects and copy the file. For example, to copy the above controlfile -
SQL > create directory D1 as '/home/pwprd/ravi';
SQL > create directory D2 as '
+PWPRD_DATA/PWPRDDR/CONTROLFILE';
SQL > exec dbms_file_transfer.copy_file('D1','
stby_PWPRD.ctl','D2', 'control01.ctl');
]

I can now see the backup and an alias created to that inside ASM.
ASMCMD [+PWPRD_DATA/PWPRDDR/CONTROLFILE] > ls -lt
Type Redund Striped Time Sys Name
N control01.ctl => +PWPRD_DATA/PWPRDDR/CONTROLFILE/Backup.324.654617039
CONTROLFILE UNPROT FINE MAY 13 14:00:00 Y Backup.324.654617039
ASMCMD [+PWPRD_DATA/PWPRDDR/CONTROLFILE] >


I change the control_files parameter in the spfile.
ALTER SYSTEM SET control_files='+PWPRD_DATA/PWPRDDR/CONTROLFILE/control01.ctl' SCOPE=SPFILE;

And simply bounce the instance. Indeed, that was rather straight.