fwodxupd - Update Omnidex keys with Eloquence replication --------------------------------------------------------- Revision: 2008-03-25 Overview -------- The Eloquence replication function replicates committed server transactions. Anything that is not maintained by the database server is outside the scope of the Eloquence replication functions. Omnidex maintains its keys outside of the database in the file system, intercepting database calls to modify index information. Consequently, any Omnidex index changes are not replicated to a slave server. This document describes the fwodxupd utility that allows updating Omnidex index information from slave-server changes. Updating is performed asynchronously from the replicated database changes but is expected to be typically close to real-time. fwodxupd utility overview ------------------------- The fwodxupd utility provides a means to update Omnidex information asynchroneously from the Eloquence forward log files. fwodxupd works by extracting information from the Eloquence forward log files and emulating a limited TurboIMAGE like environment that is sufficient to cooperate with Omnidex without actually performing a database access. The fwodxupd utility monitors the forward log files of the database server, and reconstructs equivalent database calls that follow the TurboIMAGE conventions. The fwodxupd utility requires enabling the Eloquence audit log functionality on the database server which adds additional information in the forward log files that allows reconstructing database calls. As the database server supports operations beyond the scope of the TurboIMAGE API, not all changes can be covered. Some adiministrative operations like creating or erasing a database or changing the database structure are outside the scope of the TurboIMAGE database API and require administrator intervention. Requirements ------------ The following requirements must be met: - Eloquence B.07.10 must be installed - Currently, only the HP-UX platform is supported - Forward Logging option must be enabled, the server must have sufficient disk space to hold the fw log files. - Eloquence auditing must be enabled (EnableAudit=1) - The Eloquence fwutil library must be installed (patch PE71-0803190 or superseding) Limitations ----------- The Omnidex integration in principal is unable to handle some category of database changes. Each requires manual intervention. This includes: - creating/erasing/purging a database - restoring a database - changing database structure - recovering from a backup The fwodxupd currently depends on some observed behavior of the Omnidex libraries, not on documented behavior. So it could cease to function with updated Omnidex versions or abort if the assumptions are invalid. After a system crash re-starting fwodxupd may not work as the information in the status file may be incorrect. To correct this problem, the Omnidex files need to be restored to a known point that is covered by a forward log file and then fwodxupd needs to be started from this point. After a server abort (eloqdb6 crash) the fwodxupd utility may not work correctly. Upon a server crash recovery (performed automatically on next startup) in some cases the audit information may currently be inconsistent and cause fwodxupd to fail. A kill -9 (or system crash) may currently corrupt the fwodxupd status or the Omnidex indexes as it may interrupt the processing at an arbitrary point. To correct this problem, the Omnidex files need to be restored to a known point that is covered by a forward log file and then fwodxupd needs to be started from this point. When an error condition is detected by the fwodxupd utility it aborts (writing a core dump) and writes a message to stderr. The fwodxupd utility -------------------- The fwodxupd utility is used to extract and replay transactions from an Eloquence forward log file to update Omnidex indexes. The fwodxupd needs to be executed on the same host that is managing the Omnidex files. As the fwodxupd utility reads the server forward log files it needs read access rights to the forward log files of the server. Usage: fwodxupd [options] options: -help - show usage (this list) -version - show program version information -c cfg - server configuration file -v - verbose, display progress -S - synchronize on existing log, then exit -f statfile - status file (default: fwodx.stat) -n gen - initialize from forward-log generation -I - print status file (-v to output details) The -c option is used to specify the server config file. If not present, it defaults to the default config file on the local system (/etc/opt/eloquence6/eloqdb6.cfg). By default, fwodxupd synchronizes all enqueued changes and then closely follows any on-going changes on the server. If the -S option is present, fwodxupd exits once all enqueued changes are synchronized. The fwodxupd utility uses a local file to maintain its most recent sucessfully executed call. The status file may be specified with the -f option and defaults to fwodx.stat in current directory. The -n option is used to specify the forward log file that fwodxupd should be starting with. It is required on the first use and may also be used to specify a different starting point. If not specified, the fw log file is obtained from the status file. The -n argument uses a syntax of "gen[-seg[.action]]" where gen specifies the generation of the forward log file, the optional seg specifies a file segment and the optional action specifies a specific action in this file. Typically only the generation is used. Loading the Omnidex library: The Omnidex library currently must be loaded by setting the OMNIDEX_LIB environment variable to specify the library. The SHLIB_PATH or LD_LIBRARY_PATH environment variables must be set appropriately to point to the installation directory of the Omnidex libraries and the fwodxupd image3k library _before_ the Eloquence libraries (see man dld.sl). It is important to verify the Eloquence image3k library is _not_ used with fwodxupd. You may use the -version argument to verify the fwodxupd image3k library is used. It should provide an output as below: ./fwodxupd -version fwodxupd utility 1.0 (Mar 27 2008) fwodxupd image3k library 1.0 (Mar 27 2008) FWUTIL B.07.10.06 (build Mar 19 2008) Tracing the operation: Setting the EQ3K_DEBUG environment variable to a file name (or stderr) will output some limited information on the image calls to the specified log file. The FWUTIL_DEBUG_FLAGS could be used to specify debug flags for the fwodxupd utility (eg. FWUTIL_DEBUG_FLAGS=*3). This may in some cases enable additional output. The output is written to stderr. For example: # fwodxupd -c /etc/opt/eloquence6/eloqdb6.cfg -v -S -n 4 This will run fwodxupd using the specified config file and exit once it has processed all enqueued operations. fwodxupd creates a new status file fwodx.stat in the current directory and starts from the log generation 4 (e.g fw4-1.log). # fwodxupd -c /etc/opt/eloquence6/eloqdb6.cfg -v -f fwodx.stat This will run fwodxupd using the specified config file. fwodxupd will read the specifed status file and continue operation from the most recently completed operation. Installation ------------ The installation archive includes the following files: FWODXUPD.txt - this file fwodxupd - fwodxupd utility libimage3k.sl - the fwodxupd image3k utility fwodxupd may be installed at an arbitrary location in the file system. However, the fwodxupd image3k utility may not be installed in the same location as the Eloquence libraries. initializing fwodxupd operation ------------------------------- To initialize the fwodxupd operation, the following steps are required. On the master server: 1. Create a new log generation dbctl -u dba forwardlog restart 2. Make sure that replication on the master server stops before this generation of fw log files On the slave server: 3. Rebuild the Omnidex indexes. On the master server: 4. Resume replication On the slave server: 5. Run fwodxupd from the new log generation.