This section provides an overview of the vision system software. This includes relevant behavior of the system software running on the vision system's processor board. Topics covered include software startup behavior, how to download new software, and how to interact with the vision system specific software.
When the 68332 processor board starts up, it goes through several stages. Each stage can be recognized by the information it sends over the serial port.
First the bootstrap starts up and sends the following message at 38400 baud.
BOOTSTRAP <version number>
This message will always be at 38400 baud, regardless of what baud you are running at. Therefore, if you are running at 9600 baud it will appear as some garbage characters. Do not be alarmed by this.
Next the Kernel (the operating system on the processor board) starts up, sets the baud rate to be what it was last running at (in the case of the Pioneer vision software, this will be 9600 baud, otherwise it will be 38400 baud), checksums itself, and loads custom microcode. It prints a message like the following:
Checksum OK Restoring all backups unless RAM programs are newer Microcode loaded from 17f28 to fff000, 800 bytes Anarchboot Multitasking 68332 Kernel Version 0.986 Copyright Anne Wright 1993 Compiled May 6 1996 at 15:20:43
The actual boot message will change with different versions of the kernel, but it will generally be in a similar form to that shown above.
Next it checks whether there are any programs which should be run on reset. If it detects any, it will give you an opportunity to keep them from running (see section Inhibiting Reset Run of Programs).
Checking for reset run programs Control-c to inhibit reset run...10...9...8...7...6...5...4...3...2...1...
If you did not prevent programs from being run on reset, it will start to run them at this time. First, it will initialize any libraries it finds. In the case of the software for both the stand alone and Pioneer vision system software, it should run the library `vislib-xx_x' (where `xx_x' refers to the version number of the vision library).
The vision library will beep, read the color training and persistent parameters which had been saved to the serial EEPROM on the vision system (see section Training the Vision System to See a Target Color), then beep again.
Initializing library vislib-26_2 Cognachrome Vision System Software version 26.2 [beep] Channel 0: Restored 393 bytes Channel 1: Restored 1035 bytes Channel 2: Restored 6 bytes Restored a total of 1434 bytes Restored protocol string: "$ %04xF:%04xR:%04xC#" from EEPROM [beep]
Finally it runs any application programs that are loaded. In the case of the stand alone software, the application program should be `vision-xx_x' (where `xx_x' refers to the version number). In the case of the Pioneer software, this should be `pioneer-xx_x'.
Running program vision-26_2 as process 99 vestaboot.986>
If this sequence does not proceed according as described above, determine the first step at which the actual behavior diverged.
vislib-xx_x in order for the versions
to be compatible. See section Downloading Programs to the Vision System.
When the board is reset, the kernel will normally initialize libraries and run application programs automatically (see section Startup Behavior). However, there is an opportunity to prevent it from performing this step. This is particularly useful if you are writing your own software and are putting the serial port into raw mode, or if you are using the Pioneer software and wish to prevent the vision system from entering Saphira interaction mode (which runs with the serial port in raw mode).
Do the following to inhibit programs from being run on reset:
For vision systems installed in Pioneer robots, do not confuse the reset for the vision system and the robot itself. The vision system reset button is the one labeled Boot Button on the vision interface panel (see section Pioneer Vision Interface Panel Diagram).
Checking for reset run programs Control-c to inhibit reset run...10...9...8...7...6...5...4...3...2...1...
There is only a small window of opportunity, as pressing Control-C before or after the dots will not work. If you succeed at pressing Control-C in the appropriate interval, it will say:
Reset running of programs inhibited
If nothing appears when you press reset, check the power and serial connections, and check that you are using the correct serial port (see section Troubleshooting). If you get reasonable boot messages (see section Startup Behavior), but it does not print the above message, most likely there are no programs loaded. See section Downloading Programs to the Vision System.
The behavior of the vision system is governed by the interaction mode it is in at a given time. The mode determines what processing it will perform, what information it will send over its serial port, and what commands it will respond to or ignore. The following table provides an overview of these modes. They are described in more detail in later sections.
In Tracking Upload Mode, the vision system processes and uploads tracking data. The format of the tracking data is configurable by setting the Protocol String (see section Protocol String). It will continue to upload data until it is stopped (see section Exiting Tracking Upload Mode).
You must exit this mode in order to download programs or use regular ARC commands.
In all modes except Saphira Interaction Mode, the vision system interacts with the host over a multi-streamed protocol. This allows ARC to interact simultaneously with multiple processes on the board, and present each interaction as a separate window. It also allows ARC to communicate with the board to download programs, set the baud rate, etc. (see section Downloading Programs to the Vision System).
In any mode except Saphira mode, you can download programs using the
regular download commands. If you are in Saphira mode, exit first before
downloading (see section Exiting Vision System Saphira Interaction Mode). This
section describes the minimal downloading instructions, which is all you
should need. See the ARC User's Guide for more details. It is
available at:
`http://www.newtonlabs.com/arc/arc_1.html'
The vision code is distributed in two formats:
Installation and upgrade packages for the vision system software contain
the operating system, libraries, and application code (either stand alone
or Pioneer). These are distributed as Motorola S-Record files (.s19
suffix) containing multiple executables.
These files are in a.out executable format, which is output by the
compiler. If you are writing your own programs using the full ARC
Development system, you will mainly be downloading files in this format.
WARNING: Downloading either kind of package to non-volatile memory only works for processor boards which have been specially modified to use EEPROM. All processor boards shipped with vision systems sold after March 31, 1996, are modified to use EEPROM. See section Processor Board Jumper Configuration, for more information.
To download these packages to EEPROM, start ARC and use the
kernelload command.
arc> kernelload packagename.s19
ARC will prompt you to turn off the board and hit RETURN. After you hit RETURN, turn the system back on. Downloading will begin automatically.
It is normal for ARC to pause while it reads the file, so be patient. However, if it does not start downloading after about 30 seconds, there is probably a problem. Double check that serial interaction works in general (see section Serial Port Troubleshooting).
These .s19 distribution files contain the bootstrap. This is
necessary to allow the same file to work whether downloaded by ARC or a
device programmer. As a result, when downloading using ARC, writes to the
first 16K of EEPROM will print a message that they are being skipped so as
not to overwrite the current bootstrap. This is normal, and is nothing to
worry about.
The download will fill the entire EEPROM (to address 0x40000), so it takes a while.
When the download completes, press the vision system reset button. For vision systems installed in Pioneer robots, do not confuse the reset for the vision system and for the robot itself. The vision system reset button is the one labeled Boot Button on the vision interface panel(see section Pioneer Vision Interface Panel Diagram).
Executables output by the GCC cross compiler are in a.out format. If you are writing your own programs using the full ARC Development System, you will mainly be downloading your programs in this format.
To download these files:
baud 38400
at the arc> prompt.
list command.
Programs are unloaded by using the unload filename command
at the arc> prompt.
arc> romload filename
arc> ramload filename
arc> baud 9600
run filename at the arc> prompt, or press the reset
button. For vision systems installed in Pioneer robots, do not confuse the
reset for the vision system and for the robot itself (see section Pioneer Vision Interface Panel Diagram).
In Tracking Upload Mode, the vision system will upload tracking data over stream 28 on serial port A. The format of the tracking data is configurable by setting the Protocol String (see section Protocol String). It will continue to upload data until it is stopped (see section Exiting Tracking Upload Mode).
Once in Tracking Upload Mode, do one of the following to exit:
Most other modes can be entered from TPUVIS Prompt Mode by pressing a key. See the instructions for entering the other modes for instructions.
With either the stand alone or Pioneer software, if you want to quit the
program from the TPUVIS Prompt, press capital `Q'. If you do this,
the program will no longer be running, and you will have to reset or type
run filename at the arc> prompt to run it again.
From the `tpuvis>' prompt, you can retrain the three tracking channels and change the parameters used during tracking, as well as other miscellaneous operations. To execute a command, simply type its letter (don't press return). Case is important: some commands are lowercase, while some are uppercase.
This section describes the individual `tpuvis>' commands in detail.
The border of the object is assumed to be where the distance in the color
cube from the color in the center to another color exceeds the persistent
value diffthresh (which defaults to 5000). The scale of
diffthresh is pretty arbitrary, but when it gets bigger it is more
generous about how different a color can be from the center color and still
be accepted as belonging to the same object, and when it gets smaller it
gets more stingy.
color_grow along all edges. color_grow defaults to 5.
Increasing this number increases the effect of `<' and `>', and
decreasing it decreases the effect.
In typical systems the filtering is usually set to 1. The primary problem with increasing filtering is that it increases the size of the smallest object detectable by the system. (Sometimes making targets that are wide but short can help meet a constraint of small target but large filtering). The primary advantage of increasing the filtering is that the processing time per frame is proportional to the number of transitions in the thresholded image. Increasing the filtering typically reduces the number of transitions by removing the spurious ones, which decreases the processing time per frame, which speeds up the entire system. Therefore, it is best to keep the filter level as high as you can, given the size of the smallest object you want to be able to track.
Be sure to remember the capital `S' command to save to EEPROM if you want the new filtering level to persist when you turn off the board.
When you execute this command, you will be given a choice of which protocol string to change, or if you want to modify the header: type `h' to change the header, or `a', `b', or `c' to change a particular channel. Be sure to try tracking out afterwards to see if your change had its desired effect--protocol strings can be tricky. You can use the capital letter `O' command to track a single frame; this is very useful for protocol string testing.
Remember that commands capital `S' and `R' save and restore the protocol strings as well as the colors being tracked. If you change the protocol string and are happy with it, be sure to use `S' to save it to EEPROM.
The four parameters are sony_apl, which adjusts the auto
picture level (auto gain control); sony_sharpness, which adjusts
the sharpness; sony_saturation, which adjusts the saturation; and
sony_hue, which adjusts the hue. See section Important Persistents for
more information.
Frame 70: N= 27, cg= ( 58,201) Frame 71: N= 27, cg= ( 58,202) Frame 72: N= 25, cg= ( 57,201) Frame 73: N= 27, cg= ( 59,201) Frame 74: N= 25, cg= ( 56,201)
Frame 83: N= 27, cg= ( 60,201), angle= 60, aspect ratio = 16 Frame 85: N= 28, cg= ( 57,202), angle= 53, aspect ratio = 16 Frame 87: N= 27, cg= ( 59,201), angle= 56, aspect ratio = 14 Frame 89: N= 27, cg= ( 58,202), angle= 48, aspect ratio = 15
Crosshairs are intended to be used primarily as a debugging aid. Because the timer processor generates the crosshairs in software, tracking performance will degrade somewhat when the crosshairs are on, so we recommend leaving crosshairs off during normal operation.
Also, you may notice unusual streaks to the right of the object being tracked while crosshairs are on. This doesn't indicate a problem in tracking, but is rather an artifact of minimizing crosshair's impact on tracking performace (the timer processor gives higher priority to recording the incoming tracking data, so it may be sluggish turning off the crosshair signal if there are a lot of incoming data points to record).
84: 94 -> 108 85: 88 -> 110
This section is only relevant if you are using the Pioneer software (see section Pioneer Quick Start Guide). The commands described in this section are not available if you are using the stand alone software (see section Stand Alone Quick Start Guide).
When the Pioneer software starts up, it automatically enters Saphira interaction mode (see section Startup Behavior). In this mode, the vision system runs at 9600 baud and communicates using PSOS packets--the data format used by the 6811 board on the Pioneer and the Saphira robot control program (see section Saphira Packet Protocol). In this mode, the vision system will act as a bridge between Saphira and the robot processor, handle PSOS packets from the host directed to the vision system, and upload vision data to the host formatted as PSOS packets. It will not exit this mode unless it receives a PSOS packet instructing it to. See section Exiting Vision System Saphira Interaction Mode.
The following block diagram shows the communications between the components of a vision-equipped Pioneer.
The vision system has two serial ports. In Saphira mode, one communicates with the host and the other communicates with the 6811-based Pioneer controller board. The host will typically be running the Saphira host program (the graphical robot-control environment for the Pioneer written by Kurt Konolige). You may choose instead to run ARC with the `-saphira' flag on the host (see section Starting ARC for use with the Pioneer Vision System).
PSOS packets from the host contain an address field which tells the vision system whether it should handle the packet itself, or just pass it along to the Pioneer controller. For example, commands for controlling the motors or reading the sonars are passed along to the Pioneer controller unmodified. On the other hand, commands for configuring the vision system are handled locally and are not passed on to the Pioneer controller at all.
This section discusses the commands which Saphira can send the vision system. In normal use, you should not need to issue these commands directly, as Saphira will do it for you. However, in some cases you may wish to issue these commands to the vision system directly without using Saphira. See section Interacting Directly with the Vision System using ARC, for information on setting up to communicate with the vision system in Saphira mode from ARC.
WARNING: Be sure that ARC and the Saphira host program are not both trying to talk to the robot at the same time. Exit one before starting the other.
There are several ways to make the vision system exit Saphira interaction mode:
-pioneer flag, turn on or reset the board, and
press RETURN. ARC will send a PSOS packet to the board to cause it to exit
Saphira mode.
-pioneer or -saphira
flags, issue the saphira_stop command at the arc> prompt.
ARC will send a packet to the board to cause it to exit Saphira mode.
-saphira flag, type `q' RET at
the saphira> prompt.
In the first case, the vision code will not run at all. In all of the other cases, the vision system will now be in TPUVIS Prompt Mode (see section TPUVIS Prompt Mode). In either case, regular ARC commands will work at this point (the vision system's host serial port will no longer be communicating using PSOS packets).
These commands perform operations on the tracking channels. Replace `%' with `a', `b', or `c' to specify which channel to affect. These behave the same as their counterpart commands at the TPUVIS prompt. See section Training and Configuring Channels Detail, for details. These commands are case-sensitive, so be sure to match the case listed here.
The frame commands perform a 64x48 pixel frame grab and upload information
for display. Currently the frames uploaded by these commands are displayed
using the program xv. In order to use these, be sure you have
installed xv, and that it is in your path. The default settings are
for an 8-bit display. For xv to display a frame with acceptable
quality on an 8-bit display, it must construct an appropriate 8-bit
colormap. xv creates a window with a private colormap, so move the
mouse inside the window to see what the frame really looks like (with the
mouse outside, you're likely to see some pretty strange artifacts).
Replace `#' with `0', `1', or `2' for compression amount. Currently, the only compression performed is by reducing the bits per pixel. In the future, other compression means (such as JPEG) may be incorporated.
Replace `%' with `a', `b', or `c' to specify which tracking channel the command applies to.
These commands are case-sensitive, so be sure to match the case listed here.
The vision system can analyze frame grabs to produce visual sonars data. This method calculates the distance to the nearest visible obstruction, just as sonars caculate the distance to the nearest object which reflects sound. Visual sonars can have high spatial resolution without the problems of cross-talk to which regular sonars are prone. They can also be more directional, and their speed does not depend on the maximum range as regular sonars do. They can recognize some types of obstacles that sonars miss, but can be fooled by things like changes in carpet color which would not fool regular sonars. Therefore both visual and normal sonars are useful.
The distance measurement is based on the assumption an obstacle touches the floor. In order to work, the camera must be angled towards the floor, and the floor must be a relatively solid color. First the image is processed with an edge detection filter (see section Saphira Mode Frame Commands). Then the system scans up in columns from the bottom of the image to find the vertical position in the frame of the lowest obstruction.
Obstructions are assumed to contrast with the floor. While scanning up
from the bottom, the first horizontal line in the column with a large
enough value in the edge-filtered image is assumed to belong to the nearest
obstacle. You can adjust the sensitivity of the obstacle avoidance by
setting the persistent variable edge_thresh to a value between 1 and
255. 100 is the default. Lower values increase sensitivity, and higher
values decrease sensitivity. (Persistents are viewed and set with the
`P' command, described in the next section).
These commands display the visual sonars data using the plotting program
gnuplot. A statically-linked version of gnuplot is included
with ARC for this purpose. Be sure that it is installed and in your path
to use these commands.
These commands are case-sensitive, so be sure to match the case listed here.
Persistent variables provide a method of preserving calibration variables even if the system is restarted. It is possible to view and modify any persistent variables, not just the ones particular to the Pioneer software, using the following Saphira interaction mode command. See section Persistent Variables, for more information on how to use these persistent variables. Commands are case-sensitive, so be sure to match the case listed here.
For example, to check the sensitivity of the edge detection, send `Pedge_thresh' in a PSOS packet (see section Saphira Packet Protocol). It will respond with a packet containing `edge_thresh = 100'.
Say that you now want to set the value to 200. Send `Pedge_thresh=200' in a packet. It will respond with `edge_thresh = 200 (was 100)'.
When the vision system is in Saphira mode, it is controlled using PSOS packets. This is the data format used by the 6811-based robot controller board on the Pioneer and the Saphira robot control program. Here is the exact format of the packets that the vision system accepts and generates while in Saphira mode (see section Saphira Interaction Mode).
We assume that you're already familiar with the PSOS packet protocol as described in Pioneer 1 Software Manual.
The basic operation of the vision system in Saphira mode is as follows:
whenever the host sends a packet to the vision system, if the vision
system does not recognize the packet type, the packet is forwarded to
the 6811. Whenever the 6811 sends a packet to the vision system, the
vision system forwards the packet to the host, and follows it with a
VISIONpac status packet. In this way, the vision system is
almost totally transparent between the host and the 6811 (except for
some latency).
The vision system understands two packet formats from the host:
VISION_COM and VISIONIO_COM. The former is subsumed
by the latter, and should not be used (it is retained only for backward
compatibility).
VISIONIO_COM packets contain length-prefixed ASCII strings, in the
format generated by the Saphira function sfRobotComStr. These
strings are just the Saphira commands described above (see section Saphira Interaction Mode).
#define VISIONIO_COM 100
VISIONIO_COM (= 100)
sfARGSTR (= 0x2B)
Each VISIONIO_COM command responds with a VISIONIOpac
packet when it is done. Mostly these are short status messages
(suitable for displaying to the user); they are subject to change. For now,
the main purpose of the return messages is handshaking; you should get exactly
one VISIONIOpac for each
VISIONIO_COM you send. Note that the VISIONIOpac string
is not NUL-terminated.
#define VISIONIOpac 0x80
VISIONIOpac (= 0x80)
Each time the vision system forwards a status packet from the 6811 to the
host (at approximately 10 Hz), it also sends a VISIONpac. The mode
of each of the three color channels is set by using persistent variables
(see section Persistent Variables). Each color channel can either be placed in
a line-tracking mode or one of two blob-following modes. A
VISIONpac contains some status information about the current
line-tracking settings, followed by a tracking data block for each of the
channels.
The format of the tracking data block depends on the tracking mode of the
channel. The tracking mode of the three channels are set using the
persistents pioneer_a_mode, pioneer_b_mode, and
pioneer_c_mode. Currently, the possible values for these
persistents are BLOB_MODE=0, LINE_MODE=1, and
BLOB_BB_MODE=2.
#define VISIONpac 0x50
VISIONpac (= 0x50)
line_bottom_row persistent
line_num_slices persistent
line_slice_size persistent
line_min_mass persistent
In BLOB_MODE, the center of the largest blob in image is calculated
and uploaded. If there is nothing of the tracked color in the field of
view, the data consists of all zeroes.
BLOB_MODE (= 0)
In BLOB_MODE, the center and bounding box of the largest blob in
image is calculated and uploaded. If there is nothing of the tracked color
in the field of view, the data consists of all zeroes.
In BLOB_BB_MODE, the data consists of
BLOB_BB_MODE (= 2)
To understand the LINE_MODE data chunk, we need to understand the
processing the vision system does for LINE_MODE.
In LINE_MODE, the vision system divides its field of view into
horizontal chunks called slices. Each slice has line_slice_size
rows. The bottom slice (slice 0) starts at line_bottom_row and
extends to line_bottom_row+line_slice_size. Each succeeding
slice starts line_slice_size lines higher. There are
line_num_slices slices in all.
In each slice, the vision system checks how many pixels in the slice are
of the currently-tracked color for that channel; if the number is
greater than line_min_mass, then the vision system believes it
sees the line in that slice.
LINE_MODE (= 1)
The idea is to give you an idea of what the line is doing both nearby and in the distance. You can tell what the line is doing nearby so that you can turn in the correct direction. The information about how far away you can see the line lets you slow down for curves.
In addition to the above packet types, some of the Saphira commands
return special-purpose packets. The `G#' and `T#%' commands
respond with a color frame upload (a series of FRAME24pac's), the
`E#' command responds with a grayscale frame upload (a series of
FRAME8pac's), the `D' command responds with a
DISTpac, and the `DB' command puts the system into a mode
where it sends DISTpac's continually (at somewhat less than 5
Hz).
The color and grayscale upload packets are very similar. Each frame upload splits the frame into several pieces, with some number of rows in each piece, and sends one piece in each packet. The packet consists of the following:
#define FRAME24pac 0xb0 #define FRAME8pac 0xc0
FRAME24pac (= 0xB0) or FRAME8pac (=
0xC0)
Note that frame uploads use a packet size that's slightly larger than the 200 byte maximum size mentioned in the PSOS packet documentation; this is necessary to allow an entire 64-pixel row to be uploaded in a single packet.
Currently, the compression schemes are very simple; they just throw away low-order bits. There are three compression types defined:
The remaining bits are packed somewhat oddly in a FRAME24pac. The
pixel data in a FRAME24pac always consists of a byte of red, a byte of
green, and a byte of blue, in that order. In compression mode 0 (no
compression), these are just the 1-byte values from the board. In
compression mode 1, however, each of these bytes contains the bits
from two adjacent pixels, and in compression mode 2, each byte
contains the bits from four adjacent pixels. (If the frame width is
not a multiple of 2 for mode 1, or 4 for mode 2, the frame is
effectively padded on the right with zeroes.) When data from multiple
pixels is packed into a byte, the leftmost pixel goes to the most
significant bits in the byte. The same packing scheme is used in a
FRAME8pac; of course, there's no color interleaving in that case.
Visual sonar data is sent as a DISTpac packet. It has the following
format:
#define DISTpac 0xd0
DISTpac (= 0xD0)
Each byte of the distance data is the y coordinate of the nearest
obstacle in that direction. Thus, 0 means the obstacle is far away, and
47 means that the obstacle is very close. As a special case, 255 means
that no obstacle was found at all in that direction. Currently, the
vision system sends 8 byte DISTpac's, but this might be made
configurable; it would be a good idea to make your code check the number
of bytes of data to make sure it's what you're expecting.