Administration Guide : Xinet Statistics and Database Operations — the Database, Admin subtab : About the database daemons

About the database daemons
Xinet depends on two software daemons and one program to keep information up-to-date and to communicate with MySQL:
The dblogd(8) daemon is responsible for sending information from Xinet to MySQL.
This daemon is rather like the FPO daemon that Xinet uses. The daemon automatically gets information from various programs on the file server and print-spooling system, as well as from Xinet, and then it logs this information into the database.
In the same way that Xinet generates its FPOs automatically, without the user having to initiate this process, Xinet database processing occurs automatically and in the background. Because all of the activities that are taking place on the server are happening within the integrated Xinet system, the software can help keep the server’s processing loads balanced, so work won’t slow down. This means that when your system gets busy and as a result, a lot of database processing needs to happen on the server, it won’t happen in a way that slows down your production. The database processing system will wait, letting more important interactive processes finish, and only log information at a rate that won’t overload your server. You’ll never find your server making too many FPOs at once or logging in too much data at once.
For more information about dblogd(8), refer to Scheduled syncs and the dblogd(8) manual page.
The mysqld daemon provides the Database Management System (DBMS). It’s responsible for inputting and outputting information from the database. It handles requests for accessing information that’s in the database and also responds when another daemon or program wants to write information to the database.
The syncvoltodb(8) program manually synchronizes information about files with the Xinet database.
In this section:
About the dblogd(8) daemon
The dblogd(8) daemon’s main duty is reflecting changes to the file system in the Xinet database. Here is how it happens:
1.
A change is made on an AFP volume using normal Finder actions, through Xinet’s Volumes/Users interface, by uploading, or by using ksmv(1), kscp(1), or ksrm(1).
2.
A record of the event is placed in the /var/adm/appletalk/webdblog file (or C:\Program Files\Xinet\FullPress\webdblog).
3.
The dblogd(8) daemon watches the webdblog file for new events. If it finds one, it will first determine if the event happened on a Xinet database-enabled volume. If it did, dblogd(8) will compose one or many SQL statements that will register the change in the database.
4.
The dblogd(8) daemon sends the SQL statements to mysqld, which does the actual change in the database.
After these steps, the change appears in Xinet’s database.
The dblogd(8) daemon will also run two types of “synchronizations:”
1.
The daemon will sync a directory that is browsed using Xinet where the modification times are out of date.
It will not sync any subdirectories.
2.
The dblogd(8) daemon will manage Quick Syncs.
Details about the webdblog file(s)
All events are added to webdblog by the Xinet binaries (ksd, upload, ksmv, dtool, etc).
The dblogd(8) daemon will mark events as done by setting a bit as on for each event. Events are never removed from the file. Thus, the log file will fill up with processed events over time.
The webdblog file can only hold about 1MB worth of events, which is about 10,000 events. When the log file is full, the file will be deleted. A new, empty webdblog file appears and events are added to it. So. the file gets recycled as part of daily use. It’s normal to see its file size change, growing from 0 to 1 MB in size, then returning to 0.
It’s entirely possible to have many events happen at once. This will add events to the file more quickly than dblogd(8) can process them. In that case, more log files get made. In particular:
First, the entire contents of webdblog are copied to webdblog.001.
A new, empty webdblog file is made, and new events go into it as before.
The dblogd(8) daemon will read events from the 001 file, which has the oldest events.
When it has processed all events in the 001 file, that file is removed;
The dblogd(8) daemon turns to the original webdblog file and processes those events.
If even more events are added at once, you may see webdblog.002, webdblog.003, etc. The oldest events will be in the log file with the highest number. The webdblog file will always have the most recent events. As the events are processed, the files are removed, and the oldest events are processed first.
The maximum number of webdblog files is 9999. If you have more than that, then events occur but don’t go into Xinet log files. This only happens when dblogd(8) is dead for a long time. When dblogd(8) starts, it will get all the events in the existing logs, but will not get any events that happened after those logs filled up. It’s advisable to run a syncvoltodb(8) to catch up with missing events.
The contents of the webdblog file are not text, but you can see what events are in it by running the program called printdblog.
You will find this program in /usr/etc/venture/bin on Unix systems. The printdblog program does not exist on the Windows platform.
If you need to see a particular webdblog file (say webdblog.003), run the following:
# printdblog -f webdblog.003
Running printdblog -help will show more usage options as well. For example, usage says that events that have not yet been processed will have an asterisk in front them.
Unnecessary webdblog files
Even if you are not using the database, Xinet's programs will  continue to write events to the webdblog file(s). These events will never be processed so that webdblog files will accumulate  on the server.  You can stop the creation of these webdblog files by creating an empty file called disable.venture in the appropriate directory:
Unix: /var/adm/appletalk/
Windows: C:\Program Files\Xinet\FullPress\
Afterwards, delete any remaining webdblog files.
When does dblogd(8) spawn child processes?
What does it mean when you see two or three dblogd(8) daemons running?
Normally, you will just see a single dblogd(8) whose parent is process id 1. This is the daemon process.
Every single new event found in webdblog merits a child dblogd(8). So if 100 files are added to a volume, 100 different dblogd(8) daemons will be run (one after another, not simultaneously) to process each event. They will all have different process IDs, and their parent will be the original dblogd(8).
The dblogd(8) daemon also manages periodic syncs. When such a sync is going on, a new dblogd(8) process is made that is a child of the original dblogd(8) daemon process. This process will spawn its own children if events need to be processed. So you may see a parent dblogd(8), a child, and (briefly) a grandchild. Unlike daemons called to process a single event, the dblogd(8) that manages a periodic sync will stay alive during the entire time it takes to do the synchronization.
There is one more case in which dblogd(8) is run: when a listdir runs a sync on a directory. The dblogd(8) done on a directory is done very quickly because it does not descend. Full details on this type of sync can be found in Syncs done while browsing.
States of dblogd(8) pause?
There are times when dblogd(8) does not process events from the webdblog files, for example., it has paused. Note that paused is not the same as stopped. The Summary and Settings tabs both show when dblogd(8) has stopped, meaning the process is dead. They will not show when the process has paused.
Here are some states in which you might find dblogd(8):
During a userperms sync
During a backup (Quick and Full)
During a syncvoltodb run from the command-line; dblogd(8) is paused in this case
During a periodic sync: dblogd(8) is not paused. However, since dblogd(8) manages the sync process, event processing is on hold.
A backup scheduled like a periodic sync
During a listdir sync: dblogd(8) processes events as normal (listdir puts them into webdblog).
During a database move: dblogd(8) is stopped. After the move is complete, dblogd(8) restarts. The certain way to see if dblogd(8) has paused is to read this file: /var/adm/webnative/dblogd.conf (C:\Program Files (x86)\WebNative\Admin\dblogd.conf), looking for these lines:
Paused=0
ForcePause=0
Paused will be set to 1 if dblogd(8) is paused. The ForcePause option is irrelevant.
Do any synchronizations write events to the webdblog file?
Of the four types of syncs, only one writes events to the webdblog file. The others send SQL directly to mysqld.
sync from listdir: writes to the webdblog file.
sync run periodically: writes SQL directly to mysqld
sync run from the command-line: writes SQL directly to mysqld
sync run by userperms (a submit in the Volumes/Users tab): writes SQL directly to mysqld
Running dblogd(8) in debug mode
If an event is not being processed correctly (for example, a filename is changed on the volume and the change is not reflected in Xinet), you may be asked to run dblogd(8) in debug mode. This will show us the SQL statements composed by dblogd(8) to handle the event and other debugging output.
1.
Kill dblogd(8). Use dblogd -k to kill the process.
2.
Run dblogd(8) in debug mode from a csh(1) or tcsh(1) shell:
dblogd -D >& /tmp/dblogd.out (Unix)
or
dblogd -D 2> dblogd.out (Windows)
Adding an additional -a flag redirects the debugging output to at_log(4). For example:
dblogd -D -a&
3.
4.
Stop the dblogd(8) process.
5.
Restart dblogd(8):
/usr/etc/venture/bin/dblogd (Unix)
or
C:\Program Files\Xinet\Venture\Bin\dblogd.exe (Windows)
6.
Send Xinet Technical Support (help@northplains.com) the debug file.
What to do if events are not getting into the database
Updating the database is the responsibility of dblogd(8). If changes made on the file system aren't showing up in the Xinet database, dblogd(8) is the first suspect.
Here are some possibilities to consider:
1.
Many events happened at once (adding files, removing them, etc.) and dblogd(8) has not had time to process them all.
If this is true, then you will probably see multiple webdblog files. The number of webdblog files should be declining, mysqld will be busy, and many different child dblogds should be running one at a time as each event is processed.
2.
The dblogd(8) daemon has died.
When this happens, no changes are registered at all. You can see if this is true by using:
ps -ef | grep dblogd (Unix)
or
Task Manager (Windows)
or
by looking at the Daemon page in the Xinet GUI.
3.
The dblogd(8) daemon has paused.
This happens during a periodic sync and during manual syncs.
You can verify that dblogd(8) has paused by looking at:
/var/adm/webnative/dblogd.conf (Unix)
or
C:\Program Files (x86)\WebNative\Admin\dblogd.conf
Paused will be set to 1 if dblogd(8) is paused.
4.
The dblogd(8) daemon is running, but the queries it is making are slow.
In this case, dblogd(8) is spawning children to process events as normal, but it is taking a long time for each change to register in the database. Use the mysqladmin processlist command or check the Slow Queries count in the Admin tab. It may be that a particular type of event is being processed far too slowly.
5.
The dblogd(8) daemon is suspended due to lack of disk space.
6.
The “event” is a non event, meaning it happened without a program that writes the event into webdblog. Actions done with ftp(1), carried out over SMB, or performed directly on the server are examples. If these events are part of the workflow, one can use Xinet PC Connectivity, the kscp/ksrm/ksmv commands, the venturelog(8) utility or a syncvoltodb(8) command to update the database.