Pipeline Vision Class Library (PVClib)

0.0.1

Introduction.

PVClib is a class library for building end-to-end simple computer vision systems. The goal of PVClib is to aggregate useful tools for deploying camera based CV solutions for novel user interfaces. PVClib emphasises ease of use over efficiency, and is not intended for precision applications. At this time, PVClib contains specific support for Pointgrey cameras, and contains modules for implementing a 2D pattern based table-top device tracking system. However, PVClib is designed to be customized to fit a wide range of CV tasks.

Design and the Pipeline Metaphor.

PVClib is designed around a "pipeline" abstraction. The PVClib notion of a pipeline involves a chain of processing elements exhibiting the following characteristics:

• Each processing element receives data, performs some operation and passes the data on.
• Elements are interchangeable as long as they are linked so that an output from one "pipeline plugin" matches the input of the next in the direction of information flow.
• Information flow is manifested as a series of discrete packets or frames (as in image frames) which are passed from one element to the next, undergoing data processing operations, and data format conversions at each stage.

That's pretty much as far as the pipeline metaphor extends. There is potential for a client application to expand on that metaphor to include the notion of parallel processing of pipeline elements, however no explicit support for multithreading is included in the library at this time.

Some additional characteristics of this class library include:

• Template base class system, requiring subclasses to define the input and output types for a particular "pipeline plugin" element.
• Three pipeline plugin types:
  • source pipeline plugins that are "frame producers" and therefore define an output type
  • pipeline plugins that are "frame producers" and define both input and output types
  • destination pipeline plugins that define an input type
• Single direction of information flow, from source to destination.
• Data produced by a pipeline element is the responsibility of that element, and is therefore allocated, maintained and destroyed by that element.
• Elements in the pipeline follow the "initialization is resource aquisition" paradigm.

To Do List.

PVClib is not a complete work. It is a work in progress. It is "research." Its TODO list is long, with a few standouts:

• Build the code in any environment other than win32, vc++ 6.0.
• Try using it for anything other than the initial application it was built for: A 2D tracking system that finds patterns of LEDs on a table surface.
• Implement functionality for anything other than Pointgrey cameras. (coming soon...)

Dependencies

The project comes with VC++ 6.0 project files. The following libraries may be required, in addition to the libraries included with VC++ 6.0.

• QUANTA library, available at http://www.evl.uic.edu/cavern/quanta/ - required to use the QUANTAShipper module.
• Posix threads library, included in the QUANTA distribution - required for the QUANTAShipper module.
• winsock2 library - required for the QUANTAShipper module.
• CImg image procesing library, available at http://cimg.sourceforge.net/ - required for viewer modules.
• Pointgrey FlyCapture library, included with your pointgrey camera - required for the camera module.

Usage Examples

Example 1: Hello Pointgrey.

This example demonstrates how an application, using PVClib, can display the output of a Pointgrey camera.

#include <PVC/PGRCam.h>
#include <PVC/ByteVectorViewer.h>
#include <conio.h>

#define CAMW 640     // camera width
#define CAMH 480     // camera height
#define SERN 4310174 // camera serial number

int main(int argc, char* argv[])
{

        CoordSpace2D    camSpace = {0, 0, CAMW, CAMH};
        CoordSpace2DMap oneToOne = {0, 0, CAMW, CAMH, 0, 0, CAMW, CAMH};

        PGRCam camera(camSpace, SERN); // add camera

        ByteVectorViewer viewer(&camera, oneToOne); // add viewer

        while (!_kbhit()) {

                camera.onTimer();
                viewer.onTimer();

        }

        return 0;

}

Example 2: Bright Pixel Counter.

The built in functionality that comes with PVClib is only useful for performing a specific 2D camera tracking task. In order to customize PVClib functionailty, new subclasses of the Pipeline Plugin interfaces must be implemented. The following example creates a trivial subclass of DestPipelinePlugin - one that counts pixels in an image.

The Counter class:

// Counter.h: interface for the Counter class.
//

#ifndef __COUNTER_H
#define __COUNTER_H

#include <PVC/typedefs.h>
#include <PVC/PipelinePlugin.h>

class Counter : public DestPipelinePlugin<unsigned char *>
{
public:

        Counter(FrameProducer<unsigned char *> *inputProducer, CoordSpace2D inSize, unsigned char pixThresh);
        virtual ~Counter() {};

protected:
        void mainFrame(unsigned char **inFrame);

private:
        CoordSpace2D  inputSize;
        unsigned char pixelThreshold;

};

#endif

// Counter.cpp: implementation of the Counter class.
//

#include "Counter.h"
#include <stdio.h>

Counter::Counter(FrameProducer<unsigned char *> * inputProducer, 
                                 CoordSpace2D                     inSize, 
                                 unsigned char                    pixThresh)

: DestPipelinePlugin<unsigned char *>(inputProducer), inputSize(inSize), pixelThreshold(pixThresh)
{}

void Counter::mainFrame(unsigned char **inFrame) 
{
        int count = 0;
        int width = inputSize.tr.x - inputSize.bl.x;

        for (int j=inputSize.bl.y; j < inputSize.tr.y; j++)
                for (int i=inputSize.bl.x; i < inputSize.tr.x; i++)

                        if((*inFrame)[j * width + i] > pixelThreshold)
                                count ++;

        printf("%d pixels over %d.\n", count, pixelThreshold);
}

The main program:

#include <PVC/PGRCam.h>
#include <conio.h>

#include "Counter.h"

#define CAM_W 640     // camera width
#define CAM_H 480     // camera height
#define SER_N 4310174 // camera serial number
#define THRES 128     // countable pixel thresh

int main(int argc, char* argv[])
{

        CoordSpace2D camSpace = {0, 0, CAM_W, CAM_H};

        PGRCam camera(camSpace, SER_N); // add camera

        Counter count(&camera, camSpace, THRES); // add counter

        while (!_kbhit()) {

                camera.onTimer();
                count. onTimer();

        }

        return 0;

}

Example 3: The Table Tracker.

To demonstrate a more complex application, the main loop for the table tracker is included below. Most of the code for this application resides in the pipeline processing elements, which are documented in the "Table tracking subclass layer" module.

The Counter class:

#include "QuantaShipper.h"
#include "SimCam.h"
#include "PGRCam.h"
#include "Tracker.h"
#include "ImageProc.h"
#include "Multiplexer.h"
#include "ByteVectorViewer.h"
#include "FeatureVectorViewer.h"
#include "TrackVectorViewer.h"
#include "TrackerConfig.h"
#include <vector>
#include <conio.h>

using namespace std;

#define MAX_CAMERAS 8

typedef struct {
        SourcePipelinePlugin<unsigned char *>                * camera;
        DestPipelinePlugin  <unsigned char *>                * camViewer;
        PipelinePlugin      <unsigned char *, FeatureVector> * iProc;
        DestPipelinePlugin  <FeatureVector>                  * ipViewer;
        PipelinePlugin      <FeatureVector, TrackVector>     * tracker;
        DestPipelinePlugin  <TrackVector>                    * tViewer;
} TrackSource;

extern TrackerConfig TC;

CoordSpace2DMap map_spaces(CoordSpace2D from, CoordSpace2D to);
CoordSpace2DMap equal_map (CoordSpace2D map);

int main(int argc, char* argv[])
{
        
        lua_State * L = lua_open();

        open_lua_libs(L); /* Load Lua libraries */
        add_lua_funcs(L); /* Add LambdaTracker functions */
        exec_lua_file(L); /* Execute script file */

        lua_close(L); 

        CoordSpace2D normSpace = {0, 0, 1, 1};

        TrackSource trackSources[MAX_CAMERAS];

        vector<CameraConfig>::iterator cam_i = TC.cameras.begin();

        vector<FrameProducer<TrackVector> *> muxInputs;

        for (int i=0; i < TC.cameras.size() && i < MAX_CAMERAS; i++) {

                TrackSource *TS = &trackSources[i];

                // create the camera pipeline
                TS->camera  = new PGRCam   (            cam_i->camSpace, cam_i->snum    );
                TS->iProc   = new ImageProc(TS->camera, cam_i->camSpace, cam_i->camProps);
                TS->tracker = new Tracker  (TS->iProc,  TC.devices,      cam_i->cam2Norm);
                                                                
                // add viewers
                TS->camViewer = new ByteVectorViewer   (TS->camera,  equal_map (           cam_i->camSpace));
                TS->ipViewer  = new FeatureVectorViewer(TS->iProc,   equal_map (           cam_i->camSpace));
                TS->tViewer   = new TrackVectorViewer  (TS->tracker, map_spaces(normSpace, cam_i->camSpace));

                // add to the multiplexer inputProducers vector
                muxInputs.push_back(TS->tracker);

                cam_i++;
        }

        Multiplexer multiplexer(muxInputs);

        QuantaShipper shipper(&multiplexer, TC.clients[0].addr.c_str(), TC.clients[0].port);
                
        while (!_kbhit()) {

                for (int j=0; j < TC.cameras.size() && j < MAX_CAMERAS; j++) {

                        TrackSource *TS = &trackSources[j];

                        TS->camera->   onTimer();
                        TS->iProc->    onTimer();
                        TS->tracker->  onTimer();

                        TS->camViewer->onTimer();
                        TS->ipViewer-> onTimer();
                        TS->tViewer->  onTimer();
                }

                multiplexer.onTimer();
                shipper.onTimer();

        }

        return 0;

}

CoordSpace2DMap map_spaces(CoordSpace2D from, CoordSpace2D to) {
        CoordSpace2DMap retval;
        retval.from = from;
        retval.to   = to;
        return retval;
}

CoordSpace2DMap equal_map(CoordSpace2D map) {
        CoordSpace2DMap retval;
        retval.from = map;
        retval.to   = map;
        return retval;
}

---