This page describes how to create a custom probe.

Here, we will build a probe that will track memory usage of Domestik's monitor.

1. In our case, it's not difficult to find out the process number : this probe will run within dmkcheckd process.
2. we are looking only for memory leakage due to dmkcheckd own usage, so we have to look for data and stack used

On Linux, those information can be found in /proc file-system, especially in /proc/.../status. (Notez-bien : they are more easily available in /proc/.../statm but the corresponding field is marked as broken).

Tasks to be done are the following :

1. find out our process number

It can be done by using LuaPosix getpid().

2. read /proc/<pid>/status
3. extract VmData & VmStk fields
4. build a response for dmkcheck process.

resulting to the following code :

and generating these data :

{
    'dmkcheckd' = {
        'data' = "476"
        'stack' = "136"
    }
}

Notes : in order to be compatible with process_probe

• the result is an array containing another array for which the key is dmkcheckd so it's like we are monitoring dmkcheckd using process tracking probe.
• the type of returned data is forced to process_tracking. Thanks to the line :

return res, "process_tracking"

As we want to get some trends, data have to be stored inside a database.

CREATE TABLE Domestik.probe_process_tracking (
    host TEXT NOT NULL,
    process TEXT NOT NULL,
    data INTEGER,
    stack INTEGER,
    sample_time TIMESTAMP WITH TIME ZONE
);

In addition to already known process, data and stack fields, we need

• host - on which host dmkcheckd process is running
• sample_time - when this data has been sampled

We need also to provide access to Apache's user

GRANT ALL PRIVILEGES ON TABLE Domestik.probe_process_tracking TO www;

and add some indexes to speedup access

CREATE INDEX dmkppth ON Domestik.probe_process_tracking (host);
CREATE INDEX dmkppthm ON Domestik.probe_process_tracking (host, process);

We want also a long term storage thru archiving

CREATE TABLE Domestik.probe_process_tracking_archive (
    host TEXT NOT NULL,
    process TEXT NOT NULL,
    sample_time TIMESTAMP WITH TIME ZONE NOT NULL,
    data_min REAL,
    data_max REAL,
    stack_min REAL,
    stack_max REAL,
    used_min REAL,
    used_max REAL
);

We will store both minimum and maximum value for a period of time for each value, plus a new one, used, which is the sum of data and stack (as obviously, maximum used data is not max(stack) + max(data)).

As for live data, we need also to provide access to Apache's user and to do some indexing

GRANT ALL PRIVILEGES ON TABLE Domestik.probe_process_tracking_archive TO www;

CREATE INDEX dmkpptah ON Domestik.probe_process_tracking_archive (host);
CREATE INDEX dmkpptahm ON Domestik.probe_process_tracking_archive (host, process);

This file defines a class which implements the way Domestik handles data.

It's inheriting from Probe class, an abstract class which defines the interface of each probes vs Domestik engine. Our probe class skeleton is

require_once('Domestik/Probe.inc');

class probe_process_tracking extends Probe {
}

and this class has to contains following methods

Arguments are :

• $machine - on which machine dmkcheckd process is running
• $data - sample data to store
• $sample_time - when the sample has been made

Return a boolean : true for success, false if something went wrong.

function NewSample($machine, $data, $sample_time){
        $res = true;

        foreach($data as $process => $dt){    // walk against $data
            $res = $res and $this->getContext()->dbInsert('probe_process_tracking',
                array( 'host'=>$machine->getName(), 'process'=>$process,
                    'data'=>$dt['data'], 'stack'=>$dt['stack'],
                    'sample_time'=>$time
                )
            );
        }

        return $res;
    }

Argument :

• $host - for which host we are archiving. If not provided, archive all hosts.

Compute archived data older than the day before.

Argument :

• $host - for which host we are purging. If not provided purge for all hosts.

Remove archived data older than 1 month.

Argument :

• $host - for which machine

This method returns the list of monitored process for a specific host.

function StoredData( $machine ){
		if( !( $res = $this->getContext()->dbSelect('probe_process_tracking', array('process'), array('host'=>$machine->getName()), array('process'=>true), true)))	// 0 entry or error
			return false;

		foreach( $res as $d )
			$r[] = $d['process'];

		return $r;
	}

Arguments :

• $machine - Which host
• $field - array of process name to be monitored

Please have a look .../Server/Include/Domestik/Probes/probe_process_tracking.inc for the implementation which is quite complex as we are handling several processes and not only dmkcheckd.

And the result is :

• The blue curve is the trend for data allocated by this process.
• The black curve is the trend for process stack.
• The green curve will not be generated by the code presented here but only by process_tracking probe : it corresponds to the number of occurrences of this process.

Argument :

• $host - for which machine

This method returns the list of archived process for a specific host. It's basically the same code as StoredData() but dealing archive table.

function StoredData( $machine ){
		if( !( $res = $this->getContext()->dbSelect('probe_process_tracking_archive', array('process'), array('host'=>$machine->getName()), array('process'=>true), true)))	// 0 entry or error
			return false;

		foreach( $res as $d )
			$r[] = $d['process'];

		return $r;
	}

It's more or less the same code as getStoredData() but dealing with archived data and generate ranges.