fwodxupd - Update Omnidex keys with Eloquence replication --------------------------------------------------------- Revision: 2006-05-31, Draft Overview -------- The Eloquence replication function replicates commited server transactions. Anything that is not maintained by the database server is outside the scope of the Eloquence replication functions. Omnidex maintains its keys outside of the database in the file system, intercepting database calls to modify index information. Consequently, any Omnidex index changes are not replicated to a slave server. This proposal describes an additional functionality for Eloquence to allow updating Omnidex index information from slave-server changes. Updating is performed asynchroneously from the replicated database changes but is expected to be typically close to real-time. fwodxupd utility overview ------------------------- The fwodxupd utility provides a means to update Omnidex information asynchroneously from the Eloquence forward log files. fwodxupd works by extracting information from the Eloquence forward log files and emulating a limited TurboIMAGE like environment that is sufficient to cooperate with Omnidex without actually performing a database access. The fwodxupd utility monitors the forward log files of the database server, and reconstructing equivalent database calls that follow the TurboIMAGE conventions. The fwodxupd utility requires enabling the Eloquence audit log functionality on the database server which adds additional information in the forward log files that allows reconstructing database calls. As the database server supports operations beyond the scope of the TurboIMAGE API, not all changes can be covered. Some adiministrative operations like creating or erasing a database or changing the database structure are outside the scope of the TurboIMAGE database API and require administrator intervention. Requirements ------------ The following requirements must be met: - Eloquence B.07.10 must be installed - Currently, only the HP-UX platform - Forward Logging option must be enabled, the server must have sufficient disk space to hold the fw log files. - Eloquence auditing must be enabled (EnableAudit=1) Limitations ----------- The Omnidex integration in principal is unable to handle some category of database changes. Each requires manual intervention. This includes: - creating/erasing/purging a database - restoring a database - changing database structure - recovering from a backup The fwodxupd currently depends on some observed behavior of the Omnidex libraries not on documented behavior. So it could cease to function with updated Omnidex versions or abort if the assumptions are invalid. After a system crash re-starting fwodxupd may not work as the information in the status file may be incorrect. To correct this problem, the Omnidex files need to be restored to a known point that is coreved by a forward log file and then fwodxupd needs to be started from this point. After a server abort (eloqdb6 crash) the fwodxupd utility may not work correctly. Upon a server crash recovery (performed automatically on next startup) in some cases the audit information may currently be inconsistent and cause fwodxupd to fail. To be fully functional on a replicated slave server, beta 3 of the Eloquence replication functionality is required. This includes a change to enable forward logs on the slave server. However, fwodxrepl may be tested on any recent Eloquence installation. A kill -9 may currently corrupt the fwodxupd status. To correct this problem, the Omnidex files need to be restored to a known point that is coreved by a forward log file and then fwodxupd needs to be started from this point. The error handling could be improved. Currently the fwodxupd utility will often abort (writing a core dump) and write a message to stderr if a problem is detected. The fwodxupd utility -------------------- The fwodxupd utility is used to extract and replay transactions from an Eloquence forward log file to update Omnidex indexes. The fwodxupd needs to be executed on the same host that is managing the Omnidex files. As the fwodxupd utility reads the server forward log files it needs read access rights to the forward log files of the server. Usage: fwodxupd [options] options: -help - show usage (this list) -c cfg - server configuration file -v - verbose, display progress -S - synchronize on existing log, then exit -f statfile - status file (default: fwodx.stat) -n gen - initialize from forward-log generation The -c option is used to specify the server config file. If not present, it defaults to the default config file on the local system (/etc/opt/eloquence6/eloqdb6.cfg for HP-UX and Linux). By default, fwodxupd synchronizes all enqueued changes and then closely follows any on-going changes on the server. If the -S option is present, fwodxupd exists once all enqueued changes are synchronized. The fwodxupd utility uses a local file to maintain its most recent sucessfully executed call. The status file may be specified with the -f option and defaults to fwodx.stat in current directory. The -n option is used to specify the forward log file that fwodxupd should be starting with. It is required on the first use and may also be used to specify a different starting point. If not specified, the fw log file is obtained from the status file. The -n argument uses a syntax of "gen[-seg[.action]]" where gen specifies the generation of the forward log file, the optional seg specifies a file segment and the optional action specifies a specific action in this file. Typically only the generation is used. Loading the Omnidex library: The Omnidex library currently must be loaded by setting the LD_PRELOAD environment variable (see man dld.sl). The fwodxupd utility will currently print a warning if the intrinsics are not captured (eg. the Omnidex libary is not loaded) but continue anyway (and basically just update the status file). Tracing the operation: Setting the EQ3K_DEBUG environment variable to a file name (or stderr) will output some limited information on the image calls to the specified log file. The FWUTIL_DEBUG_FLAGS could be used to specify debug flags for the fwodxupd utility (eg. FWUTIL_DEBUG_FLAGS=*3). This may in some cases enable additional output. The output is written to stderr. For example: # fwodxupd -c /etc/opt/eloquence6/eloqdb6.cfg -v -S -n 4 This will run fwodxupd using the specified config file and exit once it has processed all enqueued operations. fwodxupd creates a new status file fwodx.stat in the current directory and starts from the log generation 4 (e.g fw4-1.log). # fwodxupd -c /etc/opt/eloquence6/eloqdb6.cfg -v -f fwodx.stat This will run fwodxupd using the specified config file. fwodxupd will read the specifed status file and continue operation from the most recently completed operation. Testing fwodxupd ---------------- For testing fwodxupd the following procedures should work: - Testing on a local server: 1. Create a new log generation dbctl -u dba forwardlog restart 2. Run some updates on the database NOT loading the Omnidex library. At this point the Omnidex index is out of sync. 3. run fwodxupd from the new log generation. This should bring the Omnidex index up to date. - Testing on a replicated server: 1. Create a new log generation dbctl -u dba forwardlog restart 2. Run some updates WITH the Omnidex library loaded. If dbrepl is active, the database changes are replicated but the Omnidex changes are not. 3. On the slave server, make sure the config file specifies a locaiton and name of forward log files. 4. Manually copy the fwlog files from the master (from the fw log file created in step 1). 5. run fwodxupd from the new log generation. This should bring the Omnidex index up to date (or even ahead if dbrepl is not active).