In this Document
APPLIES TO:
Oracle GoldenGate Application Adapters - Version 12.3 and later
Information in this document applies to any platform.
GOAL
Big Data HBASE/KAFKA initial load using GoldenGate
SOLUTION
The OGG for Big Data relevant documentation is available at:
---------------
Fusion Middleware Integrating Oracle GoldenGate for Big Data
https://docs.oracle.com/goldengate/bd123110/gg-bd/GADBD/toc.htm
1.4.1.5 Initial Load Support
Replicat can already read trail files that come from both the online capture and initial load processes that write to a set of trail files. In addition, Replicat can also be configured to support the delivery of the special run initial load process using RMTTASK specification in the Extract parameter file. For more details about configuring the direct load, see Loading Data with an Oracle GoldenGate Direct Load.
5 Using the HBase Handler
https://docs.oracle.com/goldengate/bd123110/gg-bd/GADBD/using-hbase-handler.htm#GADBD397
8 Using the Kafka Handler
https://docs.oracle.com/goldengate/bd123110/gg-bd/GADBD/using-kafka-handler.htm#GADBD449
-----------
15.6 Loading Data with an Oracle GoldenGate Direct Load
https://docs.oracle.com/goldengate/c1230/gg-winux/GWUAD/instantiating-oracle-goldengate-initial-load.htm#GWUAD-GUID-0D18391E-4142-4278-82D8-2C11FB0EB4ED
+++++++++++++++++++++++++++++++++Capture Process ++++++++++++++++++++
GGSCI>ADD EXTRACT EXT, TRANLOG BEGIN NOW
GGSCI> ADD EXTTRAIL ./dirdat/aa, extract EXT, megabytes 100
EXTRACT EXT
USERID ggs, PASSWORD ggpwd
EXTTRAIL ./dirdat/aa
--use either one
LOGALLSUPPCOLS
OR
GETUPDATESBEFORES
TABLE SRC.*;
GGSCI> Start Extract EXT
Get the current SCN on the source database:
SQL> select dbms_flashback.get_system_change_number from dual;
GET_SYSTEM_CHANGE_NUMBER
------------------------
3410138433
save it and this is the SCN we are going to use to start the replicat process.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Initial load process
+++++++++++++++++++++OGG using TASK method+++++++++++++++++++++++++++
GGSCI>ADD EXTRACT <initial-load Extract>, SOURCEISTABLE
GGSCI> EDIT PARAMS <initial-load Extract>
EXTRACT INITLD
USERID ggs, PASSWORD ggpwd
RMTHOST <remotehost> , MGRPORT <remotehostport>
RMTTASK REPLICAT, GROUP rkafka
TABLE SRC.*, SQLPREDICATE 'AS OF SCN 3410138433';
GGSCI> ADD REPLICAT <initial-load Replicat>, SPECIALRUN
GGSCI> EDIT PARAMS <initial-load Replicat>
REPLICAT rkafka
setenv(OGG_HOME='/u01/app/oracle/goldengate/ogg_home/12.3.1.1/GGBD/')
TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka.props
REPORTCOUNT EVERY 1 MINUTES, RATE
GROUPTRANSOPS 10000
MAP SRC.*, TARGET KAFKAADMIN.*;
++++++++++++++++++++++++++++++OGG using FILE Method++++++++++++++++++++++++++++
GGSCI>ADD EXTRACT <initial-load Extract>, SOURCEISTABLE
SOURCEISTABLE
USERID gg, PASSWORD gg
EXTFILE /gg/dirdat/rf , maxfiles 10, megabytes 2000
or
RMTFILE /gg/dirdat/rf, maxfiles 10, megabytes 2000
TABLE SRC.*, SQLPREDICATE 'AS OF SCN 3410138433';
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Initial load replicat process
ggsci> ADD REPLICAT rkafka, EXTTRAIL /gg/dirdat/rf, NODBCHECKPOINT
REPLICAT rkafka
setenv(OGG_HOME='/u01/app/oracle/goldengate/ogg_home/12.3.1.1/GGBD/')
TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka.props
REPORTCOUNT EVERY 1 MINUTES, RATE
GROUPTRANSOPS 10000
MAP SRC.*, TARGET KAFKAADMIN.*;
Make sure the initial load is finished and there is no lag, stop the initial load replicat and delete the process.
+++++++++++++++++++++++++++++++++++Starting Incremental Replicat+++++++++++++++++++++
GGSCI> ADD REPLICAT REP, EXTTRAIL ./dirdat/aa, nodbcheckpoint
REPLICAT RPKAFKA
USERID ggs, PASSWORD ggpwd
setenv(OGG_HOME='/u01/app/oracle/goldengate/ogg_home/12.3.1.1/GGBD/')
TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka.props
REPORTCOUNT EVERY 1 MINUTES, RATE
GROUPTRANSOPS 10000
MAP SRC.*, TARGET KAFKAADMIN.*;
GGSCI> Start replicat REP, aftercsn <scn Number>
GGSCI> Start replicat REP, aftercsn 3410138433
