Robotika.cz

OSGAR

2014-07-28T00:00:00Z

John Deere X300R

Team

Milan Kroulík — project leader (CZU/TF)
Stanislav Petrásek — mechanics (CZU/TF)
Tomáš Roubíček — electronics (RobSys)
Jakub Lev — software/testing (CZU/TF)
Martin Dlouhý — software (robotika.cz)

2014

The idea is relatively old — our garden desires mowing every second week and I would not mind some „helper”. Yes, in meantime autonomous lawn mowers made serious advance, but it is not only our garden and there are much larger grass areas which require regular maintenance (castle gardens for example ).

In 2014 we also started school project, where self-driving tractor should cut grass and regularly monitor trees in a orchard. You could probably use GPS for basic navigation, but there are many obstacles (trees), which have to be avoided. You would need sensors for this task and if you have them then you can use them also for navigation.

This „orchard task” is similar to navigation in the row of maize field in Field Robot Event contest. We already got some experience there (and prices) with our robot Eduro. The very first experiment was to take Eduro, mount it on the top of the tractor and manually drive it through orchard to collect some data .

14th May, 2016 — Contest „Robot, go straight!”

The Czech season of outdoor competitions for autonomous robots begins in May in Písek. It is called Robot, go straight! (in Czech „Robotem rovně”) and autonomous robots should only navigate straight on paved road in central park. The track is 314m long and only small electrical vehicles are allowed.

In 2015 we asked organizers if it would be possible to make an exception so that we could participate with our modified garden tractor John Deere X300R. The permission was granted but we did not use it — the tractor was not ready yet.

In 2016 it looked much more promising because the school tractor was supposed to be at exhibition in Brno, one month before the contest. But there were problems with hydraulics components and this exhibition program had to be canceled.

But we wanted to push the project forward! We joked with Standa and Tomáš, that we can short safety switch under the seat and use it as emergency stop and then just put brick on the gas pedal and version 0 is ready. Well, what I did not know was that this „joke” will turn true and in reality we will need it for homologation and the first run:

homologation

1st run

4th run: out of gas

The contest is great. It runs all day (there are four runs), so we had enough time to continue to work on the robot. In the second run the robot was already driven by Python code and pedal was controlled via CAN bus. We even collected several points and „good bye” joke was in the 4th run, when the tractor run out of gas [we are so used to regularly check the battery status, but not the gas — new experience].

For more detailed story see Robotem rovně 2016 article.

4th June, 2016 — Contest „RoboOrienteering”

The contest „Robotem rovně” was great, because it got us started. Finally! But how to proceed? We were quite exhausted from the contest so we took a break next Tuesday (our group meets up once a week). And then I realized that contests are the only way how to push out project further, i.e. complete steering mechanism. The only related contest (as far as I know) was RoboOrienteering, but the dead-line was deadly. We had to finish the robot in two weeks (mainly Tomáš and Standa). The deal was set that if I can write navigation software in two weeks they will have the hardware ready. And so we fight again!

The task for RoboOrienteering 2016 was to navigate to differently evaluated orange cones (their positions are known few minutes before the start — the teams receives USB disk with configuration text file), and drop there golf balls. There are two radii: 2.5m for double and 5m for single score.

Version 0 — the simplest thing which can possibly work — was here much harder. If nothing else to score a single point it was necessary to have integrated GPS, functional ball dispenser, and also robot had to be able to avoid collision with obstacle (part of homologation).

My part was relatively easy — if you do not have any feedback data (encoders or position of steering wheel) then your program has to be very simple. My primary goal was to integrate new sensor Velodyne VLP-16, so the first tests were simple collision detection and STOP, something we wanted to have already in Písek.

Velodyne Puck VLP-16

You can have a look at video from the first outdoor test. It was not perfect. You can see „gaps” which corresponds to dropped UDP packets, and at one point you can see tractor jumping back when obstacle was detected. There was a non-trivial lag between reality and processed data.

Yes, there was a bug in (my) software. After the program start it was waiting for start button. The Velodyne socket was already opened but I started to read it after button was in on position … well, fixed, no so many things damaged, ready for competition.

The tractor first autonomously steered just few hours before the contest (we arrived in Rychnov at 1am and started testing at 5am in the parking lot. So far so good. Unfortunately none of us is expert in hydraulics and we did not know that there is too much pressure in the system. It broke during homologation (it was bit uphill so I had to increase the gas) — but over the day Standa with colleagues somehow magically managed to fix it even without proper tools. [now it is revised and working fine]

Ball dispenser

The team

Autonomous collision avoidance

Video from RoboOrienteering 2016

For more info see the article from RO16.

23rd November, 2016 — Hydromotor video

Standa sent me link to very nice descriptive video how transmission works in our John Deere tractor:

16th May, 2017 — „Robotem rovně” (again) and the first garden test

Last weekend we participated in contest Robot go straight! in Písek. It was primarily show how far we moved since the last year (brick on pedal to pass homologation). The sensor set was: odometry (encoders on front wheels with position angle of the left wheel), GPS used for reference, laser scanner SICK LMS100 and dual IP camera with wide angle lenses. For the first two runs we basically tuned the steering and collected real data of the environment. In the afternoon we added simple correction from camera data to avoid green areas (yes, this is a bit strange for lawn mower ).

The second day we tested our newly developed „cones algorithm” in the garden. For version 0 it is sufficient to prove that we can repeat given pattern 10 times (or better infinite times, but we do not want to wait so long). You can see it on the video bellow. We tried also to turn the cutting mechanism on, but surprise surprise John Deere turns it off whenever the mower is in backward motion. So we switched to a bit boring oval and did some experiments in higher grass.

Please forgive me the quality of the video — it is taken by digital camera 12 years old (which is still good for pictures but not for videos).

Robotour 2017

2017-03-08T00:00:00Z

Rules

The rules for Robotour 2017 are slightly different compared to the last years. In particular the concurrent start of all robots at given time will no longer be enforced. Exact start location will change to boundary of autonomous area, and it will be up to the teams where and when exactly they will start. The task will be to go to loading zone, where beer barrel will be manually or automatically loaded, and then deliver the payload to desired destination (unloading zone). After unloading (again manual or automatic) the robot has to leave the autonomous zone.

For each sub-task team gets one point, and extra bonus is for autonomous load/unload. In total 5 points per run/delivery request.

The detail rules are available on GitHub with tag ROBOTOUR2017: English. and Czech

Examples of roads in Žilina (Ľudovít Štúr park)

Location

The contest will take place primary in park of Ľudovít Štúr.

overview map

If you would like to somehow support this contest or you have some comments/question, please use our standard contact form.

SICK Robot Day 2016

2016-10-18T00:00:00Z

Rules in nutshell

The playing field has size approximately 7x13m. Two autonomous robots start from yellow and ping 3x3m storage area. Their task is to delivery cubes to the central 5x5m large playing field. The cubes have to be carried separately and robots get points for each covered 1x1m square. The point is gained by robot, which places the cube on given square as first one. As soon as the cube is placed it automatically becomes an obstacle (for both robots) and contact is penalized by lost of 0.5 point.

The match takes 10 minutes and every team plays two matches. The color and the oponent are randomly selected. There is RFID chip on every crossing of the field to simplify robot navigation. The code defines (X, Y) coordinates but also the „zone type”. The organizer provided teams sensors RFU 620, and the data are available over the Ethernet or CAN bus.

Results

Team	Lauf 1	Lauf 2	Bestes	Platz
Karlsuni Prag	7.0	6.0	7.0	1
Uni FR	-0.5	5.0	5.0	2
EDURO	2.0	4.5	4.5	3
Kamaro Engineering e.V.	4.0	4.0	4.0	4
Osnabrück	1.0	2.5	2.5	6
FRED	X	1.0	1.0	6
Alpaca	0.0	0.0	0.0	7
WINGmen	X	X	0.0	7
Smelý Zajko	-	-	-	-

Cogito MART (1st place)

The winner of SICK Robot Day 2016 is team Cogito MART. You can read interesting Cogito diary how the robot named Clementine was modified for the contest 12 days to the competition.

CS Freiburg (2nd place)

author: Andreas Hertle

Our robot is called Zerg based on a speedy unit (Zergling) from the strategy game Starcraft. The robot was built in 2005/2006 for a race competition, thus high velocity and acceleration are its dominant attributes.

As robotic middleware we use ROS running on Ubuntu, which for us has two major benefits:

Zerg

1. The ROS community provides a whole range of ready-to-use components, ranging from drivers for various sensors to path planning and perception libraries. Thus less time needs to be invested into the development of components not specific to the task at hand.

2. The flexible communication between ROS programs allows for easy re-configuration of the system. Multiple alternative implementations of specific components can be prepared in advance and quickly exchanged should one show significant advantages.

This year we upgraded the electronics hardware, replacing the old custom built boards with an Arduino Mega. The Arduino program reads the wheel encoders to estimate the current velocity and feeds it to the PID controller for the motors. Our robot can reach a linear velocity of 2.3 m/s, however there is rarely enough free space to utilize maximum velocity. For turning the robot relies on skid steering. In theory it can turn 360° in half a second, however that is much faster than the navigation algorithms can handle. For the competition the linear velocity was limited to more manageable 1.3 m/s and the angular to 180°/s.

Gripper

To move the foam cubes, we constructed a specialized gripper. It has a round shape so that the cubes do not need to be aligned and can be picked up at an angle. Usually at the front of the robot we put our laser range finder and to be able to see the cubes the laser sensor had to be placed less than 0.15m above ground. A solid grip can be achieved when holding the cubes equatorial, so 0.075m above ground. Furthermore, the gripper should block as few laser beams as possible in both the lowered and raised state. That is the reason for the arcing shape of the arm: the laser sensor is placed at the center.

The gripper was designed with Blender and printed on the MakerBot Replicator 5th generation. We use one Dynamixel AX12-a servos to operate the closing mechanism and two to operate the lifting mechanism. Finally, we attached a Sick WT4-2P132 distance sensor to detect whether the gripper is holding a cube.

Cube Perception

Since this year’s competition focuses on transporting foam cubes, perceiving those cubes reliably is a key component. We implemented two different approaches.

The first relies on 3d point cloud data, as produced by a Microsoft Kinect camera. In the first step the ground plane is detected and removed from the point cloud data, also points above cube size are removed. The remaining data is segmented by distance to separate individual cubes. Region growing algorithms find the visible sides of each cube. With this data the center and orientation of the cube can be computed. The point cloud also contains color data, thus the color of the cube can be determined. Unfortunately this whole process is computationally expensive. Even after subsampling the input data the algorithm requires 1 second to process a point cloud. Furthermore, the small horizontal angle (48°) of the Kinect camera and the minimum distance of 0.5m complicates the problem further.

Thus we implemented an alternative approach based on data from a laser range finder. The laser data first is segmented based on distance between two neighbouring ranges. Then we determine which point cluster are fully visible (not occluded by their neighbouring clusters). Finally, the distance between the first and last point decides whether it is a cube or something else. We can not determine the color of cube, however, the color information does not influence the behavior algorithms anyway. With this algorithm we can detect cubes in a 200° cone and between 0.05m and 10m as fast as the laser sensor produces the data (40Hz).

RFID Localization

The second half of the challenge for this competition consisted of recognizing RFID tags in the environment to distinguish the score fields and the home base field.

We modeled the location of the RFID tags as a second map layer. The first map is produced by the laser range finder. We estimate the displacement between the origin of the laser map and the rfid map with a probabilistic particle filter, similar as is done in Monte-Carlo Localization. The challenging part here was modeling the RFID sensor: the sensor produces the tag id and signal strength, which localizes the robot in a radius of up to 0.3m around that specific tag, so it is not very precise and has no orientation. Unfortunately we did not manage get the particle filter working reliably in time for the competition.

Thus, we went for an easier but less reliable method: we rely on the assumption that the enclosing fence is placed symmetrically around the RFID tags. That is the center of the enclosed area coincides with the center of the RFID map. We take the free cells in the laser map as input and compute the mean and covariance. With a Singular Value Decomposition we get the main axes and the center of the environment. Thus, we can estimate the center of the RFID map based solely on laser data.

Behavior

We encoded the behavior of the robot as a deterministic state machine. The process of placing cubes consists of the following steps:

Navigate to a random pose around the home field if no cube is detected.
Navigate to closest cube in the home field.
Lower gripper, approach cube and pick it up.
Decide where to place the cube.
Navigate to target field.
Lower gripper and release cube.

Should the cube sensor in the gripper report that no cube is in the gripper, the algorithm jumps to the appropriate state (1, 2 or 3 depending on the distance to possibly visible cubes).

The strategy is implemented in step 4. We prepared two alternatives: the aggressive strategy selects target fields closer to the opponent's home field; the defensive strategy places cubes closer to our own home field. In the competition our robot was using the defensive strategy.

A simulator was crucial during the development of the overall behavior, since we did not have the room to physically build the whole playing field in our lab. We used the Stage simulator, which simulates (mostly) in 2d and is lightweight, so that multiple robots can be simulated. The blue box in the picture represents the robot and the red boxes the cubes. The field of view of the laser sensor is displayed in green.

Challenges during the competition

From right to left: Hanna Stellmach Andreas Hertle Jens Schindler

After arriving at the Stadthalle, we soon realized that something was wrong with the robot’s main sensor, a Hokuyo UTM-30LX laser range finder. The sensor, which was working fine in our lab, kept crashing every few minutes. We figured out, that the sensor detected other infrared light sources and determined that turning itself off was the sensible thing to do. So we spent most of the preparation time getting a replacement sensor from our lab (home advantage) and mounting it on the robot. The only sensor we could get to work in the short time was a Hokuyo URG-4LX (4m max range), which is a huge downgrade from the UTM-30LX (30m max range). We assume the reduced range is to blame for failure in our first run.

We were so focussed on the problems with the laser sensor that we did not figured out the second problem before our second run: the carpet. Our robot has skid-steering but the carpet had high friction which made fine turns all but impossible.This can be observed when the robot tries to approach a target field or approach a cube. Not sure if we could have done anything about it, maybe wrap the robot’s rubber wheels in tape. Fortunately, our luck was that the sharp turn and stop maneuvers shook free the cube the robot was carrying and the cubes even rolled on empty target fields!

Additional thanks to students who did not participate in the competition but worked on components in previous semesters: Robert Grönsfeld, Natalie Prange and Hermann Ritzenthaler

EDURO Team (3rd place)

There is EDURO Team Diary when you switch this page into Czech. Hopefully there will be enough energy to create short English summary later on, but for now enjoy the videos from David/MART. See his complete SICK Robot Day 2016 video list.

Eduro "show time"

… the most cubes were placed in the very last game (not counted for Eduro)

contact form

Results

2016-09-21T00:00:00Z

Competition

Attendance

There were fourteen teams registered and twelve of them actively participated. The Czech team MART give up on Friday and also the local team WallE had some unrecoverable problems (the robot parts were immediately reused for „No! This is Patrick” team).

The Place

The Technische Hochschule Deggendorf has very nice campus on the river bank of Donau and it is surrounded by several parks. Stadthallenpark was classic park with fountain, roses and sandy roads. Then there were several long straight roads along the creek Bogenbach and river Donau. Then there was a section with metal corridor on the roof of parking garage Parkdeck Deichgarten. And if you would like to make it even harder there were bridges, up and down ramps, railway, steps … simply challenge.

Map overview

The Weather

… was just horrible. The next level would be probably snow and hail! It rained all day and the area was covered by many pools (some several centimeters deep), plus cold wind … but what does not kill you makes you stronger, so maybe it was good „push” to improve the robots and also the contest itself.

Homologation

Friday homologation was like paradise when compared to the weather contest on Saturday. It turned out that there only two teams loading the beer: Cogito used the this year option with 500ml can and Radioklub Písek with crane and full 5liter barrel. Other teams were more rational and skipped this extra bonus points/troubles (for the next years we want to discourage it a little).

Istrobotics

JECC - Fesl

Raptors

There was an exciting moment worth yellow journalism and that was then Quadron team left their robot parked (was it with STOP emergency button pressed?) their robot next to the road (and university stream) and after several minutes the robot woke up, turned sharp left and run for the water! The team members were not quick enough, but they were lucky that the stream had bushes along so they stopped the robot. The batteries were pulled out, but the robot survived.

Run way robot

Quadrons

Lois

Surprisingly something similar happened to Raptors an hour later … (but not so dramatic).

Runs

There was almost nobody for 0. run at 9am. Only Aleš from AmBot came and started test run. Other hoped for better weather and decided that there is no point to get wet if there is no punishment or points.

The official first run started at 10:30 (the even opening was at 10am), Stadthallenpark near the bridge and destination on the other side. Istrobotics was able to get first several hundred points but also AmBot, Smelý Zajko, JECC, Raptors and Quadrons moved and got some points.

The second run was at 11:30 so not much time for rest. It was on the roof of the parking garage, very windy, very wet. Here scoring teams were already at the end of the line (the best is at the end) so AmBot tried side road to avoid the high traffic. Unfortunately the robot decisions oscillated near the rusty border and did not move far. Only Istrobotics managed to get some points in the run, moving zigzag from the last position among other robots to the far end. Near the goal the robot decided to turn left but there were 3 parallel steps. Surprisingly the owners did not run after their machine and the small car loaded with the large barrel managed to get from the first step. On the second step they stopped it.

2nd run near the river bank

Cogito

The 3rd run was probably most emotive for me. All twelve robots where there, on the road parallel to railway. Here Kamaro Beteigeuze got their first points. They wanted to cross the bridge, but turned right sooner. The robot did not recognize this mistake so on the next crossing they turned right instead of left on the narrow downhill road under the bridge.

3rd run

3rd run continues

Kamaro Beteigeuze

Also Istrobotics had problem with the bridge and their robot refused to cross it. There were several collisions on the start (Radioklub Písek backed up into the opponent for example). Several teams confirmed anomalies with compass, so maybe there is high voltage cable hidden under the road like in Písek?

4th run

The 4th (last) run was from the other side of the bridge back to school. There were again all 12 robots. Team Cogito changed their strategy to move 25 meters, unload the beer can and return to start. That way they would get extra points for automatic loading, unloading and twice the points for the distance from the start, but … ARBot started the same second and was in the blind zone on the right. It looked like the racing but unfortunately at one point both robots decided to steer it towards the opponent.

Interestingly Smelý Zajko and JECC finished again on the same spot (they left the road in the same area in 1st run). Here they hit the concrete block when entering the bridge. Was it just coincidence or their robot brains using neural networks interpreted in both cases the situation the same way?

Total Score

Place	Team	1st Run	2nd Run	3rd Run	4th Run	Total
1st	Istrobotics	156	272	90	94	612
2nd	Kamaro Beteigeuze	0	0	76	94	170
3rd-5th	Smelý Zajko	28	0	0	40	68
3rd-5th	JECC - Fesl	28	0	0	40	68
3rd-5th	AmBot	66	0	0	0	66
6th	Raptors	20	0	3	0	23
7th	Cogito	0	0	0	1+5	6
8th-9th	Quadrons	1	0	0	0	1
8th-9th	ARBot	1	0	0	0	1
10th-12th	Lois	0	0	0	0	0
10th-12th	No! This is Patrick!	0	0	0	0	0
10th-12th	Radioklub Písek TCVVI	0	0	0	0	0

Conclusions

The contest was quite exhausting this year, but it was clear that there is high determination of the teams to push it further. Thanks a lot to prof. Kurpris and his colleagues — the organization was perfect! Also thanks to the sponsors and the city Deggendorf covering the accommodation and dinner expenses.

And what next? We discussed it on the Sunday workshop (as part of PAIR'16) and I would leave it to separate article, but it resonate with the wishes of teams that they do not want to enter coordinates in their robots any more (wet keyboard, unreadable papers, non-working wet notebooks, …). It is time for full autonomy.

Polish teams Quadrons and Raptors

Radioklub Písek - crane

Cogito

ARBot photos (2nd Run)

ARBot

2nd run - parking roof

AmBot and Istrobotics

Kamaro Beteigeuze

Smelý Zajko

Kamaro Betelgeuse Video

If you would like to somehow support this contest or you have some comment/query, please use our contact form.

Introduction of teams

2016-08-23T00:00:00Z

Teams

YouTube playlist of all registrations 2016

AmBot (CZ)

youtube https://youtu.be/HnTYwTIg1NQ

Robot Ferda for Robotour 2016 is a modified children's electric car ("ride-on"). The controller (based on Arduino with ATmega2560) controls the motors, utilizes magnetometer as a compass, manages three sonars to detect obstacles, reads data from an external GPS receiver and communicates with a Bluetooth converter (to respond to commands from the master system). The master system is an Android smartphone with special application (RoboNav) for GPS navigation by the map (derived from OpenStreetMap) supplemented by visual navigation according to smartphone camera (for keeping on the road).

ARBot (CZ)

youtube https://youtu.be/sW_2RUfSADQ

Robot is controlled via Zedboard with SoC Zynq 7020 manufactured by Xilinx. It has 512 MB RAM, 32 GB SD, the heart is dual-core ARM Cortex A9 + NEON at 600 MHz and it contains programmable gate array with 85 thousands units. Zedboard runs on Linux and as main programming language is used mono and C#.

The chassis is differentially driven with 3rd passive wheel. The modeller wheels have diameter 17 cm and two motors PG36555126000-50.9K with encoders are controlled by professional unit SDC2160 by Roboteq and it provides necessary traction.

There are two optical odometers ADNS 3080 for further position improvements. The next sense is „touch”. Robot has two tactile FSR sensors integrated in the front bumper.

Time-tested AHRS VN-100 from VectorNav provides information about robot orientation. The global position information is handled by GPS uBlox NEO 7M.

Robot has two sonars HC-SR04, which can be rotated via model servo motor.

The control unit for model servos is SSC-32.

The robot will use logitech c920 this year.

The energy source are 4 LiFePo cells with capacity 14.5 Ah and protected by SBM.

The chassis is built from 2mm thick plywood for air models and spruce beams 7 mm. Simply model domain. All parts were cut by laser.

Cogito (CH)

youtube https://youtu.be/sVGSWT-eOoc

She was nice and she was beauty,
she was smart and she was fine,
just in one word, she was cutie,
and her name was Clementine.

… all of it with a differentially driven platform, industrial laser scanner, home-made laser scanner, stereo camera, GPS and compass, highly modular message passing software architecture with computer vision and probabilistic localization … simply irresistible.

Istrobotics (SK)

youtube https://youtu.be/Jt5gbtevDCA

The base of the robot is modified RC model TRAXXAS E-MAXX (3903) equipped with webcamera, GPS, sonars HC-SR04, IMU with 3D compass and magnetic IRC. This year we will give a second chance to RPLIDAR. The control of the robot and basic sensors is processed via Arduino mega. Image processing and navigation handles Odroid XU4. Robot is programmed in C++ and OpenCV.

JECC (DE)

youtube https://www.youtube.com/watch?v=izHRiv0AWOg

Hardware:

Industrial COM Express computer with 6th gen. Core i7, 8GB RAM, 64GB Flash
GTX960 Graphics Card processing Deep Neural Networks and stereo vision

Custom Software using:

Qt
OpenCV
Caffe

Kamaro Beteigeuze (DE)

youtube https://youtu.be/ledYvBy0-bM

Four wheels, axle independent steering, offroad suspension. x86 PC, ARM Mikrocontrollers, CAN Bus and Ethernet. Lidar (front/rear) GPS, 9-DOF-IMU, Cameras ROS-Based Software

Lois, JECC (DE)

youtube https://www.youtube.com/watch?v=vqsAUxzY5mM

HARDWARE:

Raspberry pi 3 with Ubuntu Mate 16.04
Obstacle recognition with SICK PLS Laser-Scanner
Motor control via AVR-Mikrocontroller
Optical odometer
BNO055-Sensor

SOFTWARE:

Programmed in C/C++
Navigation via patched Navit-Software

MART (CZ)

youtube https://www.youtube.com/watch?v=6yKi9MFpmdo

The robot chassis is composed of two independent halves connected with axes. This is mainly to handle complex terrain. The wheels are driven by industrial stepper motors and each two are connected via toothed belt. The robot is differentially driven. Motors are controlled over CAN bus. Power provides two gel Pb accumulators from UPS. Robot is controlled by duo of BeagleBone Black boards. They integrated inertial unit, compass, GPS and sonars. There is a special trailer designed for barrel transportation.

No! This is Patrick! (DE)

youtube https://www.youtube.com/watch?v=AggbYBLsKwI

The main hardware consists of a Congatec TS180 COM Express module. For the main processing, it communicates with the XBox One Kinect and the Rasberry Pi and displays the crucial information on a Krämer V-800 Touchscreen.

A Raspberry Pi reads the sensor values from the GPS and the Bosch BNO055 and transmits them to the Congatec module. It receives the steering and speed values from the Congatec module, which are transmitted to the Freescale KL25Z microcontroller, which in turn controls the RC-Car model(1:5) via PWM signals.

Quadrons (PL)

youtube https://youtu.be/cQRcvG4eTOY

MECHANICS:

Quadron is a quadricycle (ultimately) autonomous mobile robot. It’s mechanical construction has the car-like kinematic configuration, which means that on the front axle there are two steering wheels, while the rear rigid axle is used to propel the vehicle (2WD). The base of the structure has become an electric quad, produced by Lifestyle-4-U GmbH. The assumption was to create universal platform, which will be used in many independent fields and disciplines. That is why we have introduced a number of electrical and mechanical modifications, especially in the supporting frame. After changes our robot gained the possibility of transporting elements of a fairly large scale, because in border cases up to 60kg, at curb weight of about 60kg. Then we adjusted the drive systems and steering mechanisms, that could be done with electric motors. For the movement of the vehicle corresponds 500W DC motor, while for the movement of the steering wheel is responsible DC gear motor and coupler system. Each engine was equipped with a driver, encoders and security features.

ELECTRONICS:

The robot is equipped with a number of sensors and modules responsible for security, location and orientation in the field. On board are:

Emergency stop
Encoders with IMU sensor to determine the position and orientation
GPS module which allows to set correct position of the vehicle
SICK 2D laser scanner and ultrasonic sensors located on the front of the vehicle which allows to see what is in front of robot
Vision system to assist control
Power supply 3x12V batteries, in order to achieve necessary voltage levels.

The concept of a multi-layer control required the use of computers at different levels. For the lower is responsible board Nucleo STM. While higher layer functions on a PC.

SOFTWARE:

To effectively manage and control the robot, we had to accept the concept of multi-layer software. We decided that the best solution would be to use platform ROS (The Robot Operating System). As a result, we have gained the ability to connect all levels. The lower is responsible for motor control through higher responsible for manual control using a wireless gamepad and simple autonomy, up to the highest layer that collects data from sensors, vision, coordinates designated route and decides where and how the vehicle is moving.

Over the top layer of completely autonomous work is still in progress, but we are at an advanced stage.

Radioklub Písek - TCVVI (CZ)

youtube https://youtu.be/HwBSFMnEKys

E-liška … is already traditional robot of Radioklub Písek. There are always some changes or rather major vehicle rebuilds. This time is Eliška driven by 4x4, with new motor in each wheel. Reworked is also independent suspension of all wheels including problematic steering (hopefully finally perfect).

The main control computer is notebook with Intel processor and static harddisk. Secondary computers for control of various groups are with ARM processor. The energy is stored in gel-lead batteries. Novelty is loading crane on which we are currently intensively working. Unfortunately new space arrangement forces us to do more change then we originally anticipated …

Raptors (PL)

youtube https://youtu.be/VuRul6QmGOs

Our rover – called Raptor, consists of about 2000 parts designed by our team. We have used AutoCad Inventor software in the design process. Our Robot consists of a couple of modules, that can work quite independently. The main module is a robot’s body, that houses almost all of the electronics and batteries. The rover is equipped with 6 wheel drive system, mounted on the rocker boogie suspension. Batteries provide energy for 90 minutes continues work and have special compartments with quick changing mechanism. Our on-board computer system is based on SB- RIO programed in Lab View environment. We are able to supervise all useful parameters concerning robot localization and orientation which make the navigation system. The navigation system runs on independent software and hardware platform. Dedicated software is written in C++ within the Robot Operating System ROS. Our system runs on multiple machines with different hardware architectures, file systems and OS but it is not a problem for ROS. Inertial data is provided by high quality Inertial Measurement Unit with six degrees of freedom, containing gyroscope and accelerometer. The sensor’s output is three axis angular velocity and linear acceleration. Using specialized filters system computes very accurate orientation. GPS module outputs position data with one meter accuracy. This data passes through the raspberry pi 3.The weight of Raptors Rover differs depending on the configuration, reaching maximum of 50 kg in the most complex version.

At our robot we use several safety systems. Due to any emergency situation, you could use din style red safety button, with cut off all active parts of rover or send emergency message from base station. Moreover we are able to fully separate batteries from any electric modules of robot, by switching off all switches located at back plate of chasse

Smely Zajko (SK)

youtube https://www.youtube.com/watch?v=PcLq-2B7rR4

HARDWARE

Parallax Motor Mount & Wheel Kit (old one) with speed encoders 2x HB-25 Motor Controller Sbot board (based on AVR ATmega128, low-level control board, hopefully soon to be replaced with STM32F103 board) Panasonic SDR T-50 camcorder Diamond Multimedia One-Touch Video Capture VC500 Hokuyo 5x SRF-08 (ultrasonic sensors) GPS NaviLock NL-302U USB SiRF III Compass with tilt compensation (HMC6343) AVR ATmega8 (compass driver) usual usb hub Power: HAZE HZS 12V 9Ah handmade wood & aluminium base red power switch and power circuitry HK6S remote control console + receiver unit for easier transport ASUS X552M PC as main controller

SOFTWARE

Ubuntu 16.04 LTS C++ app developed in Netbeans - https://github.com/Robotics-DAI-FMFI-UK /smely-zajko AVRStudio for SBot ChibiStudio for STM32F103 board FANN library for training and evaluating Neural Networks

WallE (DE)

youtube https://youtu.be/v979YcBIVLQ

The main hardware consists of a Microsoft surface tablet. For the main processing, it communicates with the XBox One Kinect and the Rasberry Pi and displays the crucial information on its main screen. A Raspberry Pi reads the sensor values from the GPS and the Bosch BNO055 and transmits them to the Microsoft Surface. It receives the steering and speed values from the Microsoft Surface, which are transmitted to the Freescale KL25Z microcontroller, which in turn controls the Continuous track model via H-Birdges.

second team of the th-deg

If you would like to somehow support this contest or you have some comments/question, please use our standard contact form.

Robotour 2016

2016-05-04T00:00:00Z

Rules

There are slight changes in rules this year: robot can automatically load and also unload at destination its payload. It will gain extra bonus points. Also there is an alternative for beer barrel — 500ml cans, if they are loaded automatically. And small note, that rule from last year the robot with the highest score starts from the last position will be used also in Robotour 2016.

The detail rules are available on GitHubu with tag ROBOTOUR2016: English, Czech

Examples of roads in Deggendorf

Location

The contest will take place in several neighbor locations, see the map.

map overview

The base will be in Technishe Hochschule Deggendorf, part of the contest in Donaupark, and also in Stadthallenpark and finally on the university campus ground.

If you would like to somehow support this contest or you have some comments/question, please use our standard contact form.

Tour the Stairs 2015

2015-09-25T00:00:00Z

Rules for year 2015

The basic rules remain the same as last year. There is an extra extension in cases when you successfully complete climbing up the stairs. The robot can then climb down and get extra points (1 step = 1 point). Note, that if robot starts to climb down in the middle of the stairs it will actually loose points!

Christian wanted extra „artists bonus points”, and I did not succeed to talk him out of it. The rules is that if more than 50% of robot body is made from the wood and the robot completes at least one step in one of the runs it will gain extra bonus 25 points. It means that this year you can gain up to 4*2*21+25=193 points.

Also note, that there is now dedicated website for this contest:

http://tour-the-stairs.com/

p.s. note, that registration form will be the same for all years and thus is available on the main page of the contest.

UPDATES

1st November 2015 — Registration video KRA Písek

I have two news for you: good one and bad one. The good news is that „Club of robotics and automatization”(KRA) from Písek fulfilled their promise and recorded their spider-like robot completing one step:

The bad news is that the registration deadline was yesterday and at the moment KRA Písek is the only registered team! If you just overlooked the deadline, then do not worry, fill the registration form and rather send me also email.

We talked with Christian, that in case like this I will prepare opponent robot …

9th November 2015 — Husky won't make it

Last week we gave a try to robot Husky — if it could (without modification) climb the stairs, … and it failed:

I talked with Chrisitian and I was quite surprised by one comment, that for some robotics is TTS contest only mechanical challenge, no intelligence is needed. Well, maybe it is time to sketch plans for 2016, when the robot will compete in loop: climb one staircase, navigate on the first floor to the second staircase, climb it down, and then back to start location. Better? Why not get ready now and practise already this year?

One more note, that on Saturday there will be also lectures in parallel to the contest, and for example Jean-Baptiste Mouret, who will talk about robot adaptation to damage, could be quite interesting .

28th November 2015 — Results and Lamia videos

Two competing teams are not many, but I think that it was still interesting show for visitors of NTK Galery. And it was funny to realize that both teams ended with the same rank:

Rank	Team	1st straight	1st spiral	2nd straight	2nd spiral	3rd straight	3rd spiral	4th straight	Total
1st/2nd	KRA Písek	4	2	5	2	5	2	6	26
1st/2nd	Lamia	2	2	3	3	3	10	3	26

Lamia - Tour the Stairs 2015 (straight)

Lamia - Tour the Stairs 2015 (twisted)

7th January 2016 — minivideo

Here are two short videos from Christian:

KRA Písek

Lamia

plus one older one …

Czech TV

ceskatelevize.cz/ivysilani/1097206490-udalosti-v-kulture

KRA Písek on ČT art

contact form

Results

2015-09-10T00:00:00Z

Competition

Attendance

There were nine teams registered and also nine teams actively participated . In reality instead of AmBot, which had serious troubles the day before contest and its participation canceled, started another German team JECC2. There was only one Robotour novice this year: team Kamaro Engineering, which we met on contest Field Robot Event 2015. To be honest I was not quite sure if they will come. But they did, their robot moved and with a bit of luck they won the contest!

The Place

It would not be the group from Písek, if they would not prepare something special. There were prepared quite challenging sections, in excuse that 10th year desires something extra. The runs were partially in parks, i.e. on park roads, but also on normal street, cobbles and surfaces with various colors. There was even cemetery, or better „reverential place”, where I was a bit uncomfortable, but there were also weddings and youngsters played there football, so the robots behaved with more respect to the deceased.

Another tricky place was a pedestrian bridge with wheelchair accessible slope to an island. And if you think that this is was it, what about passageway through house, which leads from the square to park „Palackého sady”? This is the big park you may know from Robotem rovně contest. What pleased me was the fact that no team complained. Just the opposite, several teams priced highly this city tour.

0th and 1st Run

Zero run, where we review start procedure, was surprisingly not that chaotic as previous years. The start and goal was then the same also for the first run.

We were witnesses of Murphy's law in practise — Kamaro Engineering completed the straight park road, crossed the street and find the way through the gate to cemetery. Then they choose wrong direction to avoid the tree and left the road. That was the practise run. In the real run the robot did not even start.

JECC2 went on odometry only, without GPS. The robot was only several hours old so it was nice to see that it worked reasonably well, and the full barrel was not a problem.

The winner from last year Smelý Zajko entered with new dose of courage loaded into its neural network. There was a new sensor (laser scanner Hokuyo), which kept robot away from the cemetery wall. After several minutes of hesitation with motion a little there and a little back the direction from camera finally won and in the real run Smelý Zajko managed to enter the gate, went through the place of rest and leave through small gate on the other side. Its run was terminated by an another trap: white car on the parking place. The color was classified as potential road but dark wheel were not. And so the robot oscillated and never left the white car. The next year this situation should be solved by better integration of new LIDAR.

I would mention one more luckless fellow: MarS. Also here played the camera primary part for navigation and the concrete columns colored like the road attracted the robot and caused collision.

And what about E_liška from Radioklub Písek? She went fine, but missed the gate. She flirt a while with the cemetery wall but then we were witnesses of AI in practice (carper would probably say luck only), when she replanned the road and choose an alternative around the cemetery. And she almost succeeded (the failure was near the goal).

2nd Run

The second run was recorded from the air by Dan Polák, so you can have a look:

This shots look like some mass accident on robotic highway to me. It was supposed to be a simple run, but it was not. The start was on a cycle path along river Otava, slightly curved. The direction south — so was the sun the source of problems? There were strips painted on the road, was that the problem? I do not know. In any case robots had problems and nobody completed the first curve. This means that also nobody reached the first crossing to turn left on the bridge and second junction to the island. Once a while it was adrenalin, for example when E_liška wrongly evaluated surface change and turn left. There was steep slope down to the river.

3rd Run

We decided not to trouble robots with the uphill drive to „Putimská brána” after experience from the first two runs. Instead we moved the start to the beginning of pedestrian zone along the river. Teams and the batteries had some time to recover over lunch break, and you could recognize that. Somebody would say, that this was a simple track, somebody else just the opposite.

Well there were not many ways where you could turn away: there was a wall on the left and houses on the right. But low kerb was killer for several robots. Also wide dark sewer scare off for example MarS and it turned 180 degrees from it.

sewer as an obstacle

broken wheel

NDTeam

This run was a tragedy for so far winning E_lišku from team Radioklub Písek. Due to repeated motion back and forth she broke the wheel and did not participate in the remaining runs.

This run scored NDTeam and the most points gained Kamaro Engineering. They had problem with their onboard computer so they mounted notebook on robot instead. And it worked. After first runs experiences they removed from the code procedure to return to the start … and I believe that they would not reach the goal if the code was still there. So it was the only team which managed to navigate between houses and narrow streets along the river all the way to Stone bridge. They finished 8 meters from the goal because their goal radius was set to 10m.

4th Run

The last run was on a wide street, cobblestones, uphill. These conditions did not suite in particular to MarS — small robot did not have enough power to pull the carriage with beer and small wheels turned in joint between tiles. Kamaro scored again and a lot. I expected that there are only two ways how to enter the park though houses, but they found the third one. Unfortunately it was very narrow with restaurant table on both sides so there was no chance to guess the correct direction. Even though the goal was just on the other side of the house and it was enough to convincing victory!

Total Score

Place	Team	1st Run	2nd Run	3rd Run	4th Run	Total
1.	Kamaro Engineering	0	12	385	221	618
2.	Radioklub Písek	148	74	0	-	222
3.	Smelý Zajko	122	0	3	31	156
4.	ARBot	0	28	12	86	126
5.	NDTeam	0	2	75	0	77
6.	MarS	44	0	2	0	46
7.	JECC2	26	0	0	9	35
8.	Istrobotics	4	-	17	2	23
9.	JECC	0	0	1	0	0

Workshop and PAIR'15

There was a „classic” workshop on Sunday. It was coordinated with student conference PAIR'15 (Planning in Artificial Intelligence and Robotics) this year, organized by ČVUT FEL Praha. The overlap was useful for both sides: students learned something from experiences of Robotour participants and competitors got some idea about current status of scientific research. The invited talk was about Roboauto presented by Honza Najvárek.

Technology

There were some interesting technologies used by teams this year:

Odroid (NDTeam, Istrobotics)
laser scanner vacuum cleaner (Istrobotics, Kamaro Engineering)
FPGA (ARBot)
modified BLDC motors in wheel (Radioklub písek)

Roboauto

Nice „diversification” of the competition was visit of Roboauto from Brno. You may remember them from previous years, when their robots finished on leading places. Now they experiment with modified car Hundai i40 and they presented small demo to groups of robotics in Písek. Do not worry if you missed that — there is going to be a chance to see the car on Robotem rovně 2016, where the goal is to participate and autonomously navigate through the park .

Plans for Robotour 2016

If everything goes fine then the next Robotour is going to be in Germany (Deggendorf). Note, that we also plan some small changes in the rules. In particular there will be possibility to load/unload barrel (50 points if robot pass the start line). Also 500ml cans will be allowed instead of the big barrel, but the robot has to be able to load them (5 points per can, max 10 pieces). We will keep the reverse order on start, i.e. the currently best team will start from the last position.

cameraman

Istrobotics

JECC

Thanks

I would like to thank a lot to Radioklub Písek for preparation and perfect realization of jubilee Robotour.

If you would like to somehow support this contest or you have some comment/query, please use our contact form.

Introduction of teams

2014-08-11T00:00:00Z

Teams

YouTube playlist of all registrations

AmBot (CZ)

youtube https://youtu.be/y_J8i0tHYjw

Robot Ferda is based on modified children's electric car ("ride-on") for Robotour 2015. The controller is based on Arduino with ATmega2560, controls the motors, utilizes magnetometer as a compass, manages three sonars to detect obstacles, reads data from an external GPS receiver and communicates with a Bluetooth converter (to respond to commands from the master system). This master system is an Android smartphone with proprietary application (RoboNav) for GPS navigation by the map (derived from OpenStreetMap) supplemented by visual navigation according to smartphone camera (for keeping on the road).

ARBot (CZ)

youtube https://youtu.be/DsKR7kcO96o

There are two optical odometers ADNS 3080 for further position improvements. The next sense is „touch”. Robot has two tactile FSR sensors integrated in the front bumper.

Time-tested AHRS VN-100 from VectorNav provides information about robot orientation. The global position information is handled by GPS uBlox NEO 7M.

Robot has two sonars HC-SR04, which can be rotated via model servo motor.

The control unit for model servos is SSC-32.

Robot has stereoscopy camera with chips Aptina MT9V032, which have global shutter and are able to work in HDR mode. Camera is movable in two axis via servos.

The energy source are 4 LiFePo cells with capacity 14.5 Ah and protected by SBM.

The chassis is built from 2mm thick plywood for air models and spruce beams 7 mm. Simply model domain. All parts were cut by laser.

Istrobotics (SK)

youtube https://youtu.be/rTmW52ScBX0

The robot is based on modified RC car TRAXXAS E-MAXX (3903). It is equipped with camera, GPS, sonar HC-SR04, IMU with 3D compas and magnetic IRC. Arduino mega is used for robot control and sensors reading. Odroid C1 running on linux is used for computer vision and GPS reading. Robot is programed with C++ and OpenCV.

JECC (DE)

youtube https://youtu.be/0WH-PPUnL6c

image recognition with webcam
obstacle detection with SICK PLS 101
embedded pc module with Intel Core i7
DRV8432 motor driver

Kamaro Engineering (DE)

youtube https://www.youtube.com/watch?v=txRV6iRba_w

Our Robot is completely self developed, from mechanics to software. It has four wheels, two independently steered axis, one main motor and weighs about 40kg. It was mainly developed as an agricultural robot for the Field Robot Event. The low level software is written in C++, the high level software in C# and Java. It has two LIDARs and a camera as its main sensors.

MarS

youtube https://www.youtube.com/watch?v=xDHrKi1u2z0

Robot is based on 4WD chassis of terrain RC model.

The data from GPS and compass are used for grid localization. The candidate with the highest probability is selected from the grid. Moreover it has to overweight all neighbors by given ratio to be accepted.

The algorithm for planning path from current winner position to goal is standard A*.

The road navigation is based on image processing of front camera. The odometry and GPS position then estimates the distance to crossing, which is realized by simple wall-following algorithm with compass direction correction.

NDTeam (CZ)

youtube http://youtu.be/2vXgi8JW6GU

Robot Robík:

weight about 20 kg without barrel
controlled by homemade control unit based on Cortex M3 (LPC1765)
9 DOF AHRS, GPS
power: 8S1P 5Ah LiPol
Credit card sized Linux computer Odroid U3 for video processing
OpenCV for road detection
5x sonar for obstacle detection and avoidance

Radioklub Písek (CZ)

youtube https://youtu.be/Elczrq4C2Rw

E-liška has size approximately 95x60x48 cm and weights around 40 kg. The board voltage 24V is provided by two gel-lead accumulators 12V/18Ah. E-liška has spring-loaded 4-wheels undercarriage with Ackermann steering and powered all four wheels, each with its own control unit. We replaced two back motors by BLDC for this year. We use Lidar Sick , GPS and 9 DOF innertial unit. The main control provides notebook with operating system Linux-Debian, and motor control handles own module with STM32. The power steering is addressed by own construction of H-bridges and 3phase control of BLDC motors. The main program is written in Python.

Smely Zajko (SK)

youtube http://youtu.be/aixAzwEohaI

Parallax (Motor Mount and Wheel Kit), encoders, 2xHB25 motor drivers Sbot board (based on AVR ATmega128, low-level control board) PC ASUS UL30V (main control computer) 5x SRF-08 (ultrasonic sensors) GPS NaviLock NL-302U USB SiRF III Compass with tilt compensation (HMC6343) AVR ATmega8 (compass driver) Camcorder Panasonic SDR-T50 or webcam usual usb hub Power: HAZE HZS 12V 9Ah handmade wood & aluminium base red power switch and power circuitry

If you would like to somehow support this contest or you have some comments/question, please use our standard contact form.

Robotour 2015

2015-05-25T00:00:00Z

Rules

The rules for ground vehicles are the same as previous year and are available in in PDF format in English and in Czech.

There is only one new rule regarding order at the start:

the robot with the highest score starts from the last position

Photo of potential roads/crossings

Location

The contest will take place at different places in Pisek. The bounding box is minlat='49.304811' minlon='14.1398335' maxlat='49.3107989' maxlon='14.1518927' The first start and homologation place will be near street "U Vystaviste" at 49.308258, 14.143379 (map).

If you would like to somehow support this contest or you have some comments/question, please use our standard contact form.

Husky

2014-07-29T00:00:00Z

Our robot Husky was tested for the first time on competition Robotem rovně. Here is short video of robot in motion. The source code was very simple (basically set 30% power to both motors), but still it was success — I had the robot for couple hours only.

Robot Husky

It looks like that Czech public is not very interested in this project (see results of this fandorama project) so I will try to write my experiences and test results here in English. If you have any question or comment please do not hesitate to ask (contact form is at the end of this article).

Content

2014

2015

Blog

29th July 2014 — Github

Last couple of years I prefer to program in Python. So I was very happy that Husky comes with Python wrapper. Unfortunately it looks like the focus of the company is mainly on ROS (The Robot Operating System) which we will probably use at the end too, but somehow I trust more Python. Yeah, and I do not know ROS much …

The official wrapper (for registered users only) is fine, but I miss communication logging and simple way of testing from recorded log files. Because of that I started alternative wrapper which is available on github. I still use the official wrapper as reference and it helped me yesterday to solve three bugs in generation of command packets:

length byte in message header should be smaller
there was a mistake with alignment (probably also causing the first mistake)
I was using protocol version 0 instead of version 1, which is described in the documentation

The good news is that Husky moves now and all its sensor data are properly logged and can be replayed any time after the trial. I could already observe current rising when the robot was pushing against the wall and I saw couple millimeters readings from encoders.

Well, I did not mention important „detail” that the whole platform can be controlled via serial line with command packets. You can read sensor data as response to query commands or you can set frequency how often they should be reported automatically. It reminds me CAN bus on Eduro robot, so I am quite happy with this setup .

31st July 2014 — IMU

Odometry with encoders looks fine so next step is to get compass/magnetometer working. I expected that it will be easy and that I will just request data via 0x4600 or 0x4606 message (code diff), but … no, something is wrong .

While I get response that MAX_SPEED is set to 1 m/s and MAX_ACCEL to 4 m/s2 confirmation of Magneto messages fails with response ['0x2', '0x0'], which means [1] Type not supported - The platform configuration being used does not support this command or request.?!

IMU (looks like UM6-CHR) is obviously mounted inside Husky. I wrote to CPR (Clearpath Robotics) support and got response how to test presence of IMU, but in ROS:

administratorcpr-sylv-01:~$ rostopic list
/clearpath/announce/robots
/diagnostics
/diagnostics_agg
/diagnostics_toplevel_state
/encoder
/husky/cmd_freq
/husky/cmd_vel
/husky/data/differential_output
/husky/data/differential_speed
/husky/data/encoders
/husky/data/power_status
/husky/data/safety_status
/husky/data/system_status
/husky/robot
/imu/data
/imu/mag
/imu/rpy
/imu/temperature
/inertial_ekf/odom
/joint_states
/joy
/plan_cmd_vel
/rosout
/rosout_agg
/tf
/tf_static

So you see there /imu/data and yes, if you query them via rostopic echo /imu/data you get readings like:

header: 
  seq: 6366
  stamp: 
    secs: 1406748289
    nsecs: 74127
  frame_id: imu_link
orientation: 
  x: 0.5765191582
  y: 0.4495936349
  z: 0.4151515331
  w: 0.5414056704
orientation_covariance: [0.03235951438546181, -0.022549884393811226, 
-0.028850223869085312, -0.02254987321794033, 0.019496535882353783, 
0.02177048847079277, -0.028850214555859566, 0.021770477294921875, 
0.03033282235264778]
angular_velocity: 
  x: -0.00532632599807
  y: -0.013848447595
  z: -0.00213053039923
angular_velocity_covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
linear_acceleration: 
  x: 0.013000455
  y: 1.05505101
  z: -0.095031495
linear_acceleration_covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

So IMU is connected and sends some data. So what is the problem and how it could be fixed? Now I am going to fall back to original Python wrapper if it also fails. Here is modified horizon snippet:

def platform_magnetometer(code, payload, timestamp):
  print(payload.print_format())

horizon.add_handler(platform_magnetometer, request = 'platform_magnetometer')
horizon.request_platform_magnetometer(subscription = 1)

Time to power on Husky in bedroom …

administratorcpr-sylv-01:~/md/hacking$ Traceback (most recent call last):
  File "./mag.py", line 20, in 
    horizon.request_platform_magnetometer(subscription = 1)
  File "/home/administrator/md/hacking/clearpath/horizon/__init__.py", line 449,
 in request_platform_magnetometer
    return self._protocol.request('platform_magnetometer', locals())
  File "/home/administrator/md/hacking/clearpath/horizon/protocol.py", line 297,
 in request
    self.send_message(message)
  File "/home/administrator/md/hacking/clearpath/horizon/protocol.py", line 346,
 in send_message
    raise utils.UnsupportedCodeError("Acknowledgment says Bad Code.")
clearpath.utils.UnsupportedCodeError: Acknowledgment says Bad Code.

OK, so it is not necessary my fault.

1st August 2014 — End of IMU mystery

How is it possible that I cannot see IMU on serial port when it is working in ROS?! I was going to sniff USB communication to get the answer … but I would not found it there . Any idea?

Hint:

administratorcpr-sylv-01:~$ ls /dev/ttyUSB*
/dev/ttyUSB0  /dev/ttyUSB1

Stupid me — sometimes, even for programmer, it is better to open the box instead of looking for the bug in software.

Thanks to Martin C. from CPR support: I have gotten word back from one of our software engineers. He said that since Husky doesn't have a built in IMU, only the motor and encoder messages are supported. To interface with the UM6 IMU that you have in your Husky, you can use this legacy Python module: http://aptima-ros-pkg.googlecode.com/svn/trunk/imu_um6/src/um6/driver.py

So IMU is separate module connected to separate USB port!

I still did not get nice data from IMU:

administratorcpr-sylv-01:~/md$ python driver.py
Read Pkt: BAD CHECKSUM
CMD 172 Success: True
CMD 176 Success: True
CMD 175 Success: True
rx pkt
Read Pkt: BAD CHECKSUM
rx pkt
Read Pkt: BAD CHECKSUM
rx pkt
Read Pkt: BAD CHECKSUM
rx pkt

… but I still consider it as success, for the moment.

19th August 2014 — USB devices

There are four USB devices plugged into Husky robot:

USB-serial converter for Husky platform (motors, encoders, …)
IMU unit
Logitech joystick receiver
Prime Sense Sensor (a'la Kinect)

Originally I wanted unified data source, similar to CAN bus messages, test and log each device separately and then integrate them together. Current status is not very encouraging. I can talk to Husky platform, but that is the only good news.

I can read data from joystick via pygame on Win7 (I tried also Eduro joy.py but my impression is that I am getting random buffered results from /dev/input/js0). There is a possibility to install pygame on Husky, but is it right direction?

I do not have any success talking directly to IMU. It should be /dev/clearpath/imu (symbolic link to /dev/ttyUSB0) running at 115200, but again I am getting rather some random bytes and checksums are not valid.

Regarding Prime Sense Sensor — it works on Win7 with open source drivers and I did not test it on Husky yet. That would be 2nd phase. The first task is to record and replay (drive again) recorded path.

So what to do?! One possibility is to give up low level access (I am not very comfortable with that) and use ROS as middleware. ROS seems to be working — I see data from IMU (rostopic echo imu/data) and from Joystick (rostopic echo joy). It can handle multiplex of all sources. It can also log them, probably … it is just such a monster that I still hesitate to cross that line. Again I would like small "hard coded" client, which does not need ROS library and connect to something, request data, read and parse them into Python class. Is then rospy "must have"?

20th August 2014 — ROS without ROS?

There is a light at the end of the tunnel. Maybe . I am trying to get data from existing ROS nodes, but I would like to avoid ROS installation (for Windows). I start to learn it — I had some idea, but there are tons of texts you do not really want to read … until I found nice article (short with images) on robohub.org:

http://robohub.org/ros-101-intro-to-the-robot-operating-system/

And then simple usage in

http://robohub.org/ros-101-a-practical-example/

I did not fully understand why they provide readers images for vitrual machines until I tried to install it on older machine with Ubuntu. There is no general installation it depends even on which particular version of Ubuntu you have. And the one I have is the last supported by hydro.

Recommended installation would take more than 2GB(!), which I do not have available on that old machine. So I went for Bare Bones, approx 100MB of packages, followed instructions on ROS wiki and it went fine.

I tried Hello Robot example and it worked (avoid copy and paste, or make sure you have corrected quotation marks):

martindmartind-ThinkPad-R60:~$ rostopic pub /hello std_msgs/String “Hello Robot”
ERROR: Too many arguments:
 * Given: [u'\u201cHello', u'Robot\u201d']
 * Expected: ['data']

Args are: [data]

And then I came over ROS Technical Overview. It was rather "lucky accident", but as soon as I read Most ROS users do not need to know these details... I know this is for me . This is the piece I was looking for. After simple test on Ubuntu I could connect also from Win7. See the „proof”:

Python test talking to ROS

So there is a chance …

21st August 2014 — Subscriber client

A long time ago I was thinking how to split processing power on robot Eduro. I asked my robo-friends if it is a good idea to use XMLRPC (Remote Procedure Call) to put it all together. Their answer was hard to forget: Well, then you will have TWO problems!

ROS is using XMLRPC. I am trying to create simple Subscriber client, i.e. piece of code which can receive IMU, data for example. The begging was fine — the client has to talk to the ROS Master via XMLRPC if there is any Publisher providing such information (see ROS Master API). In particular it is necessary to call registerSubscriber(caller_id, topic, topic_type, caller_api).

What are the parameters? caller_id is string ROS caller ID. Here I would guess that it can be any string describing my client node (I picked "hello_test_client"). Then there is the topic and topic_type. Because I want to test it on Hello Robot example it should be "/hello" and ""std_msgs/String", I hope. And then troublemaker caller_api. It is supposed to be string API URI of subscriber to register. Will be used for new publisher notifications.. I was surely coding this wrong, because I am getting:

[-1, 'ERROR: parameter [caller_api] is not an XMLRPC URI', 0]

Do I really have to create server in order to receive publisherUpdate remote procedure call? I am trying to write simple client, and from most (all?) examples I saw so far the clients subscribe and then wait until there is any publisher available. Is not there an option to drop this part completely? I would not mind to have there parameter blocking or timeout that if there is no publisher within given period it will fail. Why my poor client has to build also XMLRPC server just for service it does not care of?

OK, solved … see github. I am sorry for hard coded IP addresses — it is just test between PC running Ubuntu and Win7 laptop. There is hello.py which tries to subscribe to ROS Master and hello_server.py which would accept remote calls (copy from Python documentation). Note, that server does not call it immediately. But you are not allowed to fill there empty string. On the other hand "http://blablabla:123" worked too:

m:\git\husky\ros>hello.py
Publishers:
['/rosout_agg', ['/rosout']]
['/hello', ['/rostopic_2157_1408558967708']]
[1, 'Subscribed to [/hello]', ['http://martind-ThinkPad-R60:48116/']]

Now it is time to talk to the node 'http://martind-ThinkPad-R60:48116/' and convince it to open connection for me …

p.s. I did not receive any publisherUpdate even when I stopped /hello and added some sleep() … so hello_server.py side is not working properly yet.

22nd August 2014 — Hello Robot

Done. I finally convinced the other ROS node to send me (Win7 Python client) Hello Robot!

m:\git\husky\ros>hello.py
Publishers:
['/rosout_agg', ['/rosout']]
['/hello', ['/rostopic_2157_1408640226737']]
[1, 'ready on martind-ThinkPad-R60:49065', ['TCPROS', 'martind-ThinkPad-R60', 49065]]
- - - - - - - 
SIZE 174
message_definition=string data


callerid=/rostopic_2157_1408640226737
latching=1
md5sum=992ce8a1687cec8c8bd883ec73ca41d1
topic=/hello
type=std_msgs/String
- - - - - - - 
SIZE 15
Hello Robot
- - - - - - - 
Traceback (most recent call last):
  File "M:\git\husky\ros\hello.py", line 77, in 
    data = soc.recv(4)
socket.timeout: timed out

Here is the code diff — very ugly, nor parsing what I got from the node but rather patching of patches until I finally got answer. It was not that trivial as it may sound now, so let's review where I made mistakes …

The very first problem was in specification of the communication protocol — the last parameter in publisher.requestTopic(). I was using ROS Slave API as documentation and it was supposed to be: List of desired protocols for communication in order of preference.. Somewhere else I read that there are actually only two (TCPROS and UDPROS) so it took me while to realize why it is list of lists?! Never mind, at the end the documentation was correct, i.e. the last parameter is [ ["TCPROS"] ].

Great, after several unsuccessful attempt I finally got [1, 'ready on martind-ThinkPad-R60:49065', ['TCPROS', 'martind-ThinkPad-R60', 49065]], so my /hello node can be accessed via port 49065. I opened socket, connected to it, but I did not receive any data. I supposed that it is because of buffering so I played (probably wrong) with socket.TCP_NODELAY, timeouts etc. I tried to get 4 bytes of message length, guided by this nice descriptive example. Nothing.

From Technical Overview 6.1 Example it looked like client has to only connect. I should rather read TCPROS page, where there was an important information: A TCPROS subscriber is required to send the following fields.... So client has to talk to node first.

After a while I realized that there are tons of warnings in "/hello" node terminal:

Warnings from /hello node

So I added several key=value pairs with 4 bytes length prefix, random md5sum and then from warnings I got correct md5sum … yeah, very dirty way. At the end I received what I sent but still not the data . So the answer from node starts with message description (again) and the second "packet" is the real data Hello Robot string.

What I currently do not understand is why client has to specify message structure, when it first talks to the server, why it has to repeat it when it talks to publisher and finally why publisher repeats that when it talks back to subscriber.

I could not resist to do another test: instead of providing std_msgs/String I tried to send number. If you need to know how to specify number this std_msgs wiki should help. Note, that it is case sensitive, so in my case I tried rostopic pub /hello std_msgs/Int32 123. Strange thing is that when I asked server, which node provides /hello std_msgs/String I got both. I stopped the one with string and this is what I got from node:

error=topic types do not match: [std_msgs/String] vs. [std_msgs/Int32]

So I really do not understand why I have to say to ROS master what type of message I need …

TIMEOUT i.e. time to run to work …

25th August 2014 — ROS /imu/data

As soon as I got Hello Robot from /hello node I tried to receive /imu/data from Husky. I hacked hello.py and recorded received data. Now I understand that there is a big difference between std_msgs/String and message_definition=string data. The first is only a name while the other is message description. It was quite clear for /imu/data node, which provides message (topic type) std_msgs/Imu, and has many sub-elements.

So when you talk to ROS master you give it topic and only topic type (I am still not sure why), but once you start to talk to publisher you have to specify correct MD5 of the message structure. The good news if that if you do not know it the publisher responds with something like your md5 is not matching and that publisher has this md5. So you can start to talk to publisher again, this time with its MD5 … and then you get complete message description, including comment!

For std_msgs/Imu I received:

SIZE 2329
callerid=/um6_driver
latching=0
md5sum=6a62c6daae103f4ff57a132d6f95cec2
message_definition=# This is a message to hold data from an IMU (Inertial Measur
ement Unit)
#
# Accelerations should be in m/s^2 (not in g's), and rotational velocity should
be in rad/sec
#
# If the covariance of the measurement is known, it should be filled in (if all
you know is the variance of each measurement, e.g. from the datasheet, just put
those along the diagonal)
# A covariance matrix of all zeros will be interpreted as "covariance unknown",
and to use the data a covariance will have to be assumed or gotten from some oth
er source
#
# If you have no estimate for one of the data elements (e.g. your IMU doesn't pr
oduce an orientation estimate), please set element 0 of the associated covarianc
e matrix to -1
# If you are interpreting this message, please check for a value of -1 in the fi
rst element of each covariance matrix, and disregard the associated estimate.

Header header

geometry_msgs/Quaternion orientation
float64[9] orientation_covariance # Row major about x, y, z axes

geometry_msgs/Vector3 angular_velocity
float64[9] angular_velocity_covariance # Row major about x, y, z axes

geometry_msgs/Vector3 linear_acceleration
float64[9] linear_acceleration_covariance # Row major x, y z

================

MSG: std_msgs/Header
# Standard metadata for higher-level stamped data type
----
SIZE 587869811
s is generally used to communicate timestamped data
# in a particular coordinate frame.
#
# sequence ID: consecutively increasing ID uint32 seq
#Two-integer timestamp that is expressed as:
# * stamp.secs: seconds (stamp_secs) since epoch
# * stamp.nsecs: nanoseconds since stamp_secs
# time-handling sugar is provided by the client library
time stamp
#Frame this data is associated with
# 0: no frame
# 1: global frame
string frame_id

================

MSG: geometry_msgs/Quaternion
# This represents an orientation in free space in quaternion form.

float64 x
float64 y
float64 z
float64 w

================

MSG: geometry_msgs/Vector3
# This represents a vector in free space.

float64 x
float64 y
float64 z

What is surely wrong is SIZE 587869811 … but it is probably my mistake — I did not read all SIZE 2329 bytes … I guess. It should be clear from replay of log file.

Confirmed = in the log file it looks OK. Here is parsing IMU data (hard coded, for now). And here some outputs:

22227 1408728589 709250449 8
imu_link
(-0.007217399499999999, -0.7315086163, -0.6816246364999999, -0.015408308699999998)

22228 1408728589 760128957 8
imu_link
(-0.0073516766999999995, -0.7315086163, -0.6816246364999999, -0.015475447299999999)

22229 1408728589 811034145 8
imu_link
(-0.007586661799999999, -0.7314414776999999, -0.6816582057999999, -0.015676863099999997)

It is nice that every message contains sequential number and timestamp. What I am not sure is frame_id in particular with comment 0: no frame, 1: global frame when I received string imu_link …

27th August 2014 — Hello ROS Family

OK, now I have hacked whole ROS family: subscriber, publisher and XMLRPC server (for details see github). It is still quite dirty, but I can receive hello string and send it. The next step would be to finish parsing of other Husky messages, and try to publish command to move. To be honest I am not very happy about it — I am thinking about crazy hybrid, where sensors would be taken from ROS while I would talk to the platform directly, logging all currently unnecessary data, like currents and temperatures on both motors.

p.s. I did not found overall ROS Husky messages documentation … so probably the simplest thing is to look for description files directly on the platform.

p.s.2 Husky moved! with pseudo-ROS-Python code . I searched for *.msg in /org/ros/hydro directory. There are plenty of them and the one I needed was geometry_msgs/Twist which contains two vectors: one for linear and one for angular velocity. I am not sure if it is good to commit another IP-comment-hacking patch … but without that it would not be clear what I did … so here is another tunnel digging to Husky ROS code.

29th August 2014 — node.py

I am currently trying to put all bits I learned from /hello experiments together and create ROS node which will subscribe to several topics and publish at least one (send drive command) — see node.py. At the moment it is logging couple Husky ROS topics, each communication into separate file, and it creates also metalog recording order of all received messages.

I decided to use stupid single-threaded solution (based on KISS principle). Note, that there is in reality second thread handling XMLRPC server, but I would ignore it for the moment. The timeouts are set to 0.0 so it is actively pulling all TCP connections — not nice, but the machine has to do something anyway. I may change it later to couple milliseconds. If you read Python documentation you would hope for socket.timeout exception. Well, I was also naive … so you get socket.error instead and it is different on Windows and Linux. It is this code:

try:
    data = self.readFn( 4096 )
except socket.timeout as e:
    print e # it should contain partial data            
except socket.error as (errno, errStr):
    assert errno in [10035,11], (errno, errStr) 
       # Windows 'A non-blocking socket operation could not be completed immediately'
       # Linux (11, 'Resource temporarily unavailable')
    data = ""

As I see it now it is surely wrong as socket.timeout does not handle received data. For timeout exception may already read couple bytes and there is no other way how to pass them to data. But I did not get timeout yet (OK, to be sure I added assert there). Now I would guess that you either get some bytes i.e. no timeout or there is nothing ready and thus you get socket.error.

Other than that it looks fine, except there is a partially expected issue with buffering: when I run node.py remotely from Win7 the metalog looks something like this:

…
/husky/data/encoders
/husky/data/encoders
/husky/data/safety_status
/husky/data/safety_status
/imu/data
/imu/data
/imu/data
/imu/data
/husky/data/power_status
/husky/data/encoders
/husky/data/encoders
/husky/data/safety_status
/husky/data/safety_status
/imu/data
/imu/data
…

but on Husky it is fine:

…
/imu/data
/husky/data/encoders
/imu/data
/husky/data/safety_status
/husky/data/power_status
/imu/data
/husky/data/encoders
/imu/data
/husky/data/safety_status
/imu/data
/husky/data/encoders
/imu/data 
…

2nd September 2014 — replay metalog

Finally! Last week I made a stupid mistake and could not find it — I forgot to call self.master.registerPublisher in order to get contacts to clients. Now this is fixed and node.py now also provides simple topic publishing (see gihub diff).

There is also a newer version, where node.py accepts name of metalog (instead of master and node IPs) and then you can replay whole run. This morning it was tested only on publish /hello and subscribe /hello, so I am looking forward to do it on Husky (with publishing velocity command).

3rd September 2014 — Heartbeat

Today I moved with Husky couple centimeters using NodeROS.update() (see code diff). Sure enough there were some bugs, and one was related to failure of 100% reproducibility, i.e. major bug (later found in update() function).

The older version of update() checks all subscribers and repeats that in cycle until at least one sends data. The problem is that sometimes only one subscriber sends the data while next time it can be three subscribers, for example. And it is not clear from metalog which update got which topics updated. It is now fixed with extra '—' separator … not very scientific, but clear what is going on. The metalog now looks like this:

_imu_data140903_064426.log
_husky_data_encoders140903_064426.log
_husky_data_power_status140903_064426.log
_husky_data_safety_status140903_064426.log
_joy140903_064426.log
_husky_data_system_status140903_064427.log
_husky_cmd_vel140903_064427.log
/husky/cmd_vel
/imu/data
/husky/data/encoders
/husky/data/power_status
—
/husky/cmd_vel
/imu/data
/husky/data/safety_status
/husky/data/encoders
—
/husky/cmd_vel
/imu/data
/husky/data/safety_status
/husky/data/encoders
— 
…

And what is the heartbeat from today's title? Well, not all topics are equal and not all of them are frequent and regular. I need one, which is regular and which will dictate robot frequency of control. In the case of Husky it is /husky/data/encoders — update() is not finished until there is at least one update of encoders is received.

4th September 2014 — Joyride

I started new class HuskyROS to experiment how difficult it is to use my „node”. One of the first experiments was to drive forward when you hold GREEN button and terminate program when you press RED (see code).

It moved but it was shaking a lot. Why? Well, on boot up there is another colliding process husky_joystick … so the first step was to kill it.

Now I was seriously moving — instead of 10cm test in bedroom it moved 2m in the entrance hall . But it was not turning … so I changed the ROS message

# This expresses velocity in free space broken into its linear and angular parts.
Vector3  linear
Vector3  angular

to this:

return struct.pack("dddddd", speed,0,0, angularSpeed,angularSpeed,angularSpeed)

… then it turned for sure. The correct solution is on github.

7th September 2014 — Quaternions and loose wheel/axis

What I like on robotics is that it is quite varied. One moment you try to learn quaternions and the very next moment you have to deal with broken wheel. It is not computer science only and it is also not mechanics only — it is a bit of everything. So let's start from the beginning …

This weekend I took Husky for outdoor testing, finally. I started with two simple functions goStraight() and turn(). And I ended there too. For the moment I ignore signs so it is always going forward and always turning left. The data from encoders, when going straight, were reasonable — I took average of left and right traveled distance, and moved slowly (0.1 m/s) so I ignored speed ramps … and maybe the system takes care of that??

The troubles come as soon as I was turning. I did not try that much at home, you know carpet, and even in the garden it made "nice" holes in the lawn. At first I tried to turn 90 degrees based on odometry data. I was quite naive. It is basically the difference of left and right traveled distance, but to get the angle you also need robot width … or better distance between left and right steering wheel. Husky has 4 wheels, 2+2 connected with a chain, so that have to skid. And they do. So once I set WIDTH to measured 55cm it turned only approximately 40 degrees (instead of 90) [it was probably skidding diagonally so the distance could be 1m instead].

Then there was much more serious problem: the left front wheel looked a bit (approximately 1cm) pulled off and you could wiggle it a bit. I was not so big deal when going straight, but it was stopper for turning. There was no other way than to open the robot.

I must say, that the Husky robot is very nice done. Extra plus was that all I needed was a set of hexagonal wrenches. And the problem? It looks like there was missing lock on that particular wheel axis and thus it was no longer in the bearing. It is possible that I did something wrong when loading/unloading the robot to/from a car, but … it should be a bit industrial and survive. There was no lock laying around so it is possible that it is missing for a long time …

Quaternions — it is quite a while when I worked with them, and I forgot everything. Why do I need them? Well, the robot orientation in /imu/data is defined by Quaternion. If you do not know what it is then have a look at wikipedia: Quaterinion and conversion to Euler angles. After some messing around (you can tell from commented lines in IMU code) I found the magic formula:

self.imu =  math.atan2(2*(q0*q1+q2*q3), 1-2*(q1*q1+q2*q2)) # along X axis

After that Husky turned close to 90 degrees once, but like 130 degrees next time. It is probably integration of compass, gyros and accelerators … but I am not sure.

My last experiment was with /imu/mag, which is really the direct source I want.

Compass readings for more than one complete rotation

There is one outlier reading but other than that it looks almost circular and center is close to (0,0). Some calibration/transformation would improve it, but it could be good for the first record & replay.

After that Husky stopped to talk to me. The weather was very nice today, so maybe the processor case was getting too hot?? Maybe there is another ROS topic describing it??

One more note, that setting velocity command (0,0) actually switches to neutral and robot will start to move if it is on a small slope. So is there some command for active braking?

The lock ring should be from inside!

Opened front of Husky

Test field

11th September 2014 — Goodbye Husky!

Husky went home yesterday night, back to France. There are still some unresolved issues, like the snap ring for the front left wheel, but there was some progress anyway. I was in touch with Martin from Clearpath Robotics and I learned that there is a jumper on motor controller which defines default behavior: "coast" mode or "brake" mode. Our Husky had it in "coast" mode, so when I send command (0,0) robot would move downhill by itself.

It is necessary to open both front and back cover in order to change the jumpers. I did that and find out that one connector for temperature measurements (my guess, 3-PIN connector with resistor loop) was loose. I did not know how to put it back, so I started to open front cover again (it is 8 screws + 2 big ones for the bumper). In the middle of the operation I realized that I took a picture of the motor controller before I changed the jumper, so I could copy that:

Jumper and 3-PIN cable on motor controller

The bad news is that there is no way how to control braking from the software. You have to choose. If you are in the terrain braking is surely the choice — I rather carried Husky uphill because without the brake it could be fatal disaster. On the other hand if you want to just push the robot around, like when you load it to the car, you want "coast" mode. Note, that pushing will generate enough power that robot will start braking (even if you removed the battery and it is in E-stop mode).

What else? Probably the PrimeSense sensor. Sylvio bought it a year ago, but then the company was bought by Apple (see the discussion of upset users). The websites www.primesense.com and openni.org are no longer working. There is a source code on github, but it probably won't be supported any more (click on the title link, for example).

ROS was supposed to support Kinect and PrimeSense, and after reading several websites and almost corrupting robot installation I gave up. No ROS. An alternative seems to be new OpenNI 2.0 driver available in beta stage: http://structure.io/openni. When I downloaded it for windows it work immediately. Also on Linux I could use NiViewer, but Husky does not have monitor so you get freeglut (./NiViewer): failed to open display … note, that this was already good news! The others were like /opt/ros/hydro/lib/openni_camera/openni_node: symbol lookup error: /opt/ros/hydro/lib/libnodeletlib.so: undefined symbol: _ZN3ros7console5printEPNS0_10FilterBaseEPvNS0_6levels5LevelEPKciS7_S7_z or Open failed: USB interface is not supported!. Note important choice is 32 vs. 64bit installation — the other will not work (Husky required 64bit, Sylvio notebook 32bit).

OK, so I was able to theoretically run some program, but how to get the images and depth data? It turned out that there is small Python wrapper already available: https://pypi.python.org/pypi/primesense and although the short example had a small mistake (get_sensor_info() requires some parameter, probably sensor index), I got first Husky depth data . If you need image use dev.create_color_stream(). Now I have two raw arrays as last memory of Husky …

Is this „The End”? No, this is just the beginning! A friend of mine added ssh tunnel in start-up script, so whenever is Husky connected to the internet I could work on it. [Yes, I did not mention this detail — you can plug in Ethernet cable in the WiFi router and then you get both: old Access Point and robot connected to the Internet]. So that was the motivation to get the PrimeSense working. Now I could get at least some reference pictures what is/was going on 1000km away … could be fun … or nightmare .

3rd February 2015 — Husky at CZU

Today I finally spent the afternoon playing with Sylvio's robot Husky. It is parked now at the university (CZU — Czech University of Life Sciences Prague) for the winter (?). The hardest part was to recover mine almost half a year old memories, even with the help of this blog .

Question: where is the starting point, what was the last used script?

Answer: the best version worked with ROS, i.e. ros/huskyros.py. It has two parameters (IP of the ROS master and ROS client).

Question: how to get images/depth from PrimeSense?

Answer: this part was not in github but was still left on Husky. There is python wrapper and I was able to get the data with initial version of openni2/getpic.py.

Note, that there is a bug in current getpic.py script. I wanted to use gziped data, but if you write array with 16bit values the write() works for normal file but not for gzip. TODO. Now I have collected data but reader reject them that the file size does not match the real size:

File "C:\Python27\lib\gzip.py", line 349, in _read_eof
    raise IOError, "Incorrect length of data produced"
IOError: Incorrect length of data produced

And how was Husky driving? Well, not much and very shaky. Why? I realized that when I was going home … there was another controller running since bootup and I forgot to kill it. Next time, hopefully with processing of depth data.

10th February 2015 — Simple Depth Viewer

Yesterday I completed simple depth openni2/viewer.py. In particular I fixed reading 16bit distance array. This simple script displays the row data from PrimeSense. The decision if the file contains depth data or color data is based on its size only. Expected is 320x240 array and depth has 2 bytes per pixel while color has 3 bytes. There was necessary some mirroring and swapping colors. Also note, that depth in millimeters is divided by 20 and uint8 value is used in gray scale image (i.e. resolution 2cm and max distance 5m). Finally value 0 probably means missing information, so it is ignored for minimal distance.

Depth auto-portrait

There is also a fix of writing gzipped data — I could repeat the bug (?) with numpy 16bit array (see diff), and then fix it with array of bytes (diff). I hope that this will help today .

The next step is to integrate this distance sensor with driving. Prepared is sketch of followme.py for collection of some real data. This robot behaviour was very popular on robot Eduro laser scanner — see video or source code eduro/followme.py.

11th February 2015 — To the Wall

Thare was a big contrast between the afternoon and the evening. I exchanged few mails with Ruud about 100 nodes ROS system and later that day I was trying to get Husky to move with PrimeSense sensor (few ROS nodes). While the OpenNI2 driver with a viewer is working fine and fast on Win7, I was not able to get ROS node working half year ago. I could try upgrade, but I would not be sure that it would work after that at all. At the end I was glad for at least slowly working sample in Python. Sigh.

So where are we now? Husky moves and collects pictures and depth data from the PrimeSense sensor (see diff). If there is an obstacle within 1 meter it stops. The motion is very slow (0.1m/s) because also the data collection is extremely slow. I hacked logging of two threads communication (in this sense if Ruud quite right, that Python is not robotic middleware and I do not have clean ready-to-go classes for communication, logging, replay etc.) and it takes in average 30 cycles = 3 seconds!? Well, but at least we started.

Surprises? If I ignore my stupidity with typo of Husky IP address at the beginning — it has to be on fixed network 192.168.1.x and I wrote 192.162.1.x so no wonder it did not want to connect. There was another IP issue: If I run HuskyROS with parameters localhost localhost it worked but not for followme.py!? Desperately I tried 127.0.0.1 and that worked in both cases — no idea why :-(.

Finally when I was driving in the school corridor the distance estimation sometimes crashed:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "./followme.py", line 32, in run
    self.minDist = centerArea[mask].min()/1000.0
ValueError: zero-size array to minimum.reduce without identity

This was easy to fix. I used mask for non-zero numpy elements (I was glad that numpy is installed on Husky machine) and if there was too much free space it was looking for minimum of empty array. The same fix was also necessary in the viewer (diff).

Jakub via PrimeSense

Distance image

TODO list:

sync depth and color data collection
try Process instead of Thread
detect human and floor
navigate to free space
follow human
outdoor test

12th February 2015 — depth_stream.start()

I feel little bit stupid now. I was convinced that 32-bit OpenNI2 driver did not work on my laptop, but … did I really tried that? Was not I dealing with Linux on Husky?? Python 2.7 32bit did not work with 64bit DLL, which is not surprising. When I run 64bit Python 2.7 the getpic.py worked without any modification.

Now I installed also 32bit version from structure.io/openni and it still did not work … for a while. And now it works. Both 32bit as well as 64bit version. I believe that Windows did not release the OpenNI2.dll from the memory (???) so I was still using 64bit version even the file was already replaced.

And it works as slow as on Linux .

So my apologize to Linux driver — it is the depth_stream.start() which is very slow. Once you start it and only grab data with depth_stream.read_frame() then it is much faster (approx 10Hz). Here is the diff.

If you would like to read a bit about the sensor technology have a look at this older article.

24th February 2015 — followme.py ver0

There is version 0 of followme.py (hash):

The code is relatively simple. It scans depth image approximately at 3Hz, selects the nearest point in given ROI and turns that direction with speed proportional to the distance.

There is TODO for version 1 — narrow search area centered around previously detected nearest object (similar to Eduro version). You can see that the red circle (nearest point) sometimes jumps to wall or some other nearby objects.

p.s. this video was created by viewer.py.

23rd April 2015 — go.py into sun

I has been a while again. There were school lectures and I was playing with Katarina so hardly any progress with Husky :-(. The weather was great two days ago, so I finally did some outdoor tests with PrimeSense sensor. I wrote very simple go.py script: if there is an obstacle on the left, turn right. If the obstacle is on the right, turn left. If it is very close STOP and otherwise go slowly forward. I see that Here is the code core:

if prev is None or prev < safeDist:
    robot.setSpeedPxPa( 0, 0 )
elif prev < safeDist*2:
    # turn in place
    if obstacleDir > 0: # i.e. on the left
        robot.setSpeedPxPa( 0.0, math.radians(-10) )
    else:
        robot.setSpeedPxPa( 0.0, math.radians(10) )
else:
    robot.setSpeedPxPa( maxSpeed, 0 )

It was so simple that it worked within 10 minutes (yes, it was mostly copy and paste from folowme.py) with logging so that I could travel back and forth the corridor and then I could go outside. I expected that it will not work at all in the sun, but it was not that bad. But the sun light is surely limiting which you can see from this short video log file:

The red circle is the nearest obstacle and when approaching the corner you get nice oscillations. But other than that it moved fine.

The logs from outside in the bright sun are mostly black = no depth information at all. On the other hand it was enough if somebody walked close, cast a shadow and in that the distance was measured.

13th May 2015 — Husky & Heidi (first test)

I am preparing demo for the competition Robot go Straight!, which will take place in Písek this weekend (16th May 2015). This time I would like to show combination with ARDrone2 Heidi. The idea is that Heidi would fly approximately 1 meter above the Husky robot and stream video as navigation data for the ground platform. The plan is to reuse code heidi/rr_drone.py from last year.

Yesterday I was finally happy with navigation above oriented roundel — see guide/lesson7.py. I plan to change the configuration from down pointing camera to front (ARDrone2 API supports detection of oriented roundel even if you do not use the down pointing camera for video streaming). Heidi's part is ready for „integration testing”.

The troubles were with Husky. I repeated go.py test (using emergency STOP button to pause the robot) and 3 times it rebooted. The battery was probably on the edge, but … I added battery status (diff) so now in robot.power is fraction of charge based on ROS /husky/data/power_status. This value started at 0.37, with occasional drop to 0.25, but the first computer restart happened at 0.30. In the second run it rebooted at 0.22 where I expected that it is still safe to use it. At the moment I am not sure if it was battery issue — if yes, I may have troubles because the battery does not charge quickly if no then I do not know :-(.

My next plan is to properly handle Emergency STOP. It is OK for stopping, but if you unblock it and there was some former move command Husky „jumps” … yes, that could be another reason for reboot even in the second case I made sure that there was an obstacle in front of the robot so it would first send (0,0) motion command.

The first and only video is not very exiting (Husky did not move), so at least one picture from Heidi's down pointing camera after takeoff:

Husky as landing zone

14th May 2015 — 2 days to RR2015

Well, it will be hard. Robotem Rovne 2015 is going to start in 48hours and I do not have properly working version 0. Moreover Husky was making some bad jokes yesterday: when I started the go.py with eStop pressed after unblocking it reboot the internal PC?! Repeatably (3x). It worked when eStop was not pressed, but then it „jumped” after release. I added small pause in the code to send 0 velocity command even after eStop was released (diff), but it is hard to tell if that helped (I have Husky at home and there is really no space to move around). I was so desperate that I wrote to CPR support and I would like to thank Martin C. for his quick response . If nothing else it was good that there is somebody I can talk to about the issue …

...the commands received during estop are not buffered, and Husky will always act on the latest commands received. The timeout still applies. This means that if the commands are stopped before the e-stop is released, the Husky shouldn't move. If the commands are continued, the Husky will move at the full speed commanded once the e-stop is released."

… which is something I am doing now, so we will see during further testing. It could be related to some other issues — I am putting landing zone as lid of the Husky, maybe the PrimeSense is taking too much power, or the battery?? We will see.

This morning I tried to install OpenCV via Install-OpenCV.git but it failed due to:

The following packages will be upgraded:
  libglib2.0-0 libjpeg-turbo8 libpixman-1-0
3 upgraded, 137 newly installed, 0 to remove and 243 not upgraded.

… at least I suppose that this was the problem and not

WARNING: The following packages cannot be authenticated!
  libglib2.0-0 libatk1.0-data libatk1.0-0 libavutil51 libgsm1 liborc-0.4-0
…

So I tried to upgrade, which is surely not good idea in such a short remaining time, and hit another „wall” (at least for me, unexperienced Linux/Ubuntu user):

$ sudo apt-get upgrade
Reading package lists… Done
Building dependency tree
Reading state information… Done
You might want to run 'apt-get -f install' to correct these.
The following packages have unmet dependencies:
 ros-hydro-roscpp : 
Depends: 
ros-hydro-cpp-common (>= 0.3.17) but 0.3.16-0precise-20130919-0111-+0000 is installed
Depends: 
ros-hydro-roscpp-traits (>= 0.3.17) but 0.3.16-0precise-20130919-0231-+0000 is installed
E: Unmet dependencies. Try using -f.

Using force does not sound good to me, so there will be no OpenCV for RR2015 and I will probably run drone part from laptop.

TO BE CONTINUED

15th May 2015 — H+H Network (failure)

Last night and this morning I tried to setup Heidi+Husky network, and I did not succeeded yet. According to document From The Desk of The Robotsmiths: The Husky may be connected to the Internet in two ways. The first is by connecting the WAN port on the Microhard radio to a live Internet connection. The second is by logging in to the Microhard via wifi, and changing its SSID and wifi password to match that of an existing wifi network in your location.

Year ago I used the first case and it worked, although Sylvio had problems probably due to colliding IP addresses. The ARDrone2 talks over Wi-Fi, so this time I need to use the second option. I found manual for similar router (in Husky is VIP2-2400) and there I found what I needed, i.e. that I have to change Access Point (An Access Point may provide a wireless data connection to many clients, such as stations, repeaters, or other supported wireless devices such as laptops etc.) to Station/Client (A Station may sustain one wireless connection, i.e. to an Access Point.). I could fill SSID of our home Wi-Fi with password and the connection was established. I could see nice colors with signal bar, but … I could not ping outside (Destination Net Unreachable).

With the drone it was similar or rather worse. Router IP 192.168.1.1 is colliding with the drone IP, so first I changed that to 192.168.1.13. I also tried to change Gateway to Router, but in my case it is like monkey trying to do some programming :-(. So far no luck.

18th May 2015 — The Contest Robotem Rovně 2015

The contest is over. If you want to jump to pictures or results, feel free …

There is an hour for robot homologation (8am-9am) followed by four competition runs. The task is simple — just go straight 314 meters without leaving the park road (see interview about the competition history in English). This year there were 25 competing robots. Moreover new „specials” sub-category was created where flying, walking, omni-directional and other non-standard robots were separately evaluated.

What a pain! We arrived with both robots (Husky + Heidi) shortly after the big tent for participants was build. I started go.py, which I was using already for several weeks, and:

administratorcpr-sylv-01:~/md/ros$ ./go.py 127.0.0.1 127.0.0.1
Go straight …
METALOG: logs/meta_150516_020733.log
STARTING /node_test_ros at http://127.0.0.1:8000
Listening on port 8000 …
Traceback (most recent call last):
  File "./go.py", line 157, in 
    go( metalog, assertWrite, ipPair )
  File "./go.py", line 80, in go
    robot = HuskyROS( filename=metalog.getLog("node"), replay=metalog.replay, ipPair=ipPair )
  File "/home/administrator/md/ros/huskyros.py", line 33, in __init__
    filename=filename, replay=replay, assertWrite=assertWrite )
  File "/home/administrator/md/ros/node.py", line 90, in __init__
    logStream = LoggedStream( self.requestTopic( topic ).recv, prefix=topic.replace('/','_') )
  File "/home/administrator/md/ros/node.py", line 111, in requestTopic
    assert len(publishers) == 1, publishers # i.e. fails if publisher is not ready now
AssertionError: []

What?! This never happened to me so I started again, and again … still the same result. I was bit in panic mode, rebooted the Husky … the same again. When I calmed down I modified the assert (diff) so I know what publisher is failing.

AssertionError: ('/imu/data', [])

OK, I can live without IMU, at least for the homologation. Then I remembered that I can read ROS data directly from console, so

administratorcpr-sylv-01:~/git/husky/ros$ rostopic echo /imu/data
ERROR: Cannot load message class for [std_msgs/Imu]. Are your messages built?

??? The command is maybe wrong, but it does not matter now. It did not help.

After reboot I got:

administratorcpr-sylv-01:~$ rostopic echo /imu/data
WARNING: topic [/imu/data] does not appear to be published yet

[WARN] [WallTime: 1431759925.748272] Inbound TCP/IP connection failed:
connection from sender terminated before handshake header received. 0 bytes
were received. Please check sender for additional details.  

[WARN] [WallTime: 1431759926.051634] Inbound TCP/IP connection failed:
connection from sender terminated before handshake header received. 0 bytes
were received. Please check sender for additional details.  

[WARN] [WallTime: 1431759926.361188] Inbound TCP/IP connection failed:
connection from sender terminated before handshake header received. 0 bytes
were received. Please check sender for additional details.

…

Nice. I removed IMU from my code. Guess what? Another assert with publisher failure. This time with motor controller and there is no way to drive Husky without motor controller. Dead End.

I tried simple Python code (without ROS) from last year:

administratorcpr-sylv-01:~/md$ ./rr.py
Exception in thread clearpath.horizon.transports.Serial.Receiver:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/opt/ros/hydro/lib/python2.7/dist-packages/clearpath/horizon/transports.py", 
line 465, in run
    message = self._get_message()
  File "/opt/ros/hydro/lib/python2.7/dist-packages/clearpath/horizon/transports.py", 
line 536, in _get_message
    chars = self._serial.read(1000)
  File "/usr/lib/python2.7/dist-packages/serial/serialposix.py", line 449, in read
    buf = os.read(self.fd, size-len(read))
OSError: [Errno 11] Resource temporarily unavailable

Traceback (most recent call last):
  File "./rr.py", line 28, in 
    horizon.set_differential_output(30, 30)
  File "/opt/ros/hydro/lib/python2.7/dist-packages/clearpath/horizon/__init__.py", 
line 244, in set_differential_output
    self._protocol.command('differential_output', locals())
  File "/opt/ros/hydro/lib/python2.7/dist-packages/clearpath/horizon/protocol.py", 
line 287, in command
    self.send_message(messages.Message.command(name, args, self.timestamp(), 
no_ack=(not self.acks)))
  File "/opt/ros/hydro/lib/python2.7/dist-packages/clearpath/horizon/protocol.py", 
line 372, in send_message
    raise utils.TimeoutError("Message Timeout Occurred!")
clearpath.utils.TimeoutError: Message Timeout Occurred!

I expected that some other running process is using /dev/ttyUSB1 so my program cannot open it, but … surprise there was no /dev/ttyUSB1!!! I had like 5 minutes remaining for the homologation so I took Heidi, tried rr_drone.py from last year, left the road and hit the lamp in 2 meters from start. Never-the-less I was listed at the end, after all homologated robots, so I could still compete …

So where is /dev/ttyUSB1?! I unplugged and plugged IMU and motor controller USB cables and that was the major breakthrough. My luck was back

administratorcpr-sylv-01:~$ ls /dev/ttyU*
ls: cannot access /dev/ttyU*: No such file or directory

administratorcpr-sylv-01:~$ ls /dev/ttyU*
/dev/ttyUSB0  /dev/ttyUSB1

I do not know how to restart all failing ROS nodes and after reboot the USB devices disappeared again so I focused on direct COM and my former program (diff) and again I was getting strange reports:

0x8010
Traceback (most recent call last):
  File "./husky.py", line 180, in 
    testRR2015( com )
  File "./husky.py", line 154, in testRR2015
    robot.update()
  File "./husky.py", line 83, in update
    timestamp, msgType, data = self.readPacket()
  File "./husky.py", line 57, in readPacket
    length = ord(self.com.read(1))
  File "/home/administrator/md/logit.py", line 32, in read
    s = self._com.read( numChars )
  File "/usr/lib/python2.7/dist-packages/serial/serialposix.py", line 449, in read
    buf = os.read(self.fd, size-len(read))
OSError: [Errno 11] Resource temporarily unavailable

0x8200
0x8202
0x8800
Traceback (most recent call last):
  File "./husky.py", line 180, in 
    testRR2015( com )
  File "./husky.py", line 154, in testRR2015
    robot.update()
  File "./husky.py", line 83, in update
    timestamp, msgType, data = self.readPacket()
  File "./husky.py", line 59, in readPacket
    assert length+notLength == 0xFF, (length, notLength)
AssertionError: (242, 83)

The Clearpath Python wrapper probably hides this, because ./rr.py worked.

Husky was announced for the first run. I was so thrown off by the Husky behavior that I did not give Heidi enough attention. I started the drone from my notebook, then unblocked eStop to run old code on Husky and then the show began. It was exciting until Heidi landed between moving Husky wheels! I know immediately what it was — the time was 60s after start … yes, I had still testing version for Heidi (diff also with changes after 2nd run) with timeout set to 60s. Pity, but I hit eStop fast enough so Heidi was still as one part.

In the second run Heidi already had much longer time for the hoverAboveRoundel() task. It only went bit too high and after approximately 10m (on the first crossing) lost Husky. Yes, there was no coordination of both robots (I may describe dropping off functionality list at the end of this). After that I increased the speed and aggression of corrections. I also changed camera view to front camera as it may collect more interesting images.

In the third round I warned the spectators about parameter changes (without any chance to test them) and yes, the drone was much more aggressive. But it was also much more interesting to watch. As in previous runs if the target (landing platform with oriented roundel) was not detected the drone moved slowly up. When detection worked for last 100 pictures (the bottom camera runs at 60Hz) then it was moving down to 1 meter above the roundel. As far as I remember both robots cooperated in this run and it was terminated due to Husky leaving road.

For the final round I modified Husky code. Yeah, it was big change (diff) … instead of horizon.set_differential_output(30, 30) I called horizon.set_differential_output(30, 33) and the robot moved almost straight. I had to stop it not because of the leaving the road but because of queue of people in front of one kiosk. I was not complaining, because I lost Heidi somewhere in the middle of the way so 90 meters/points was more than what I could hope for.

Timeout - so here is at least video from the first run. Note, that the drone starts with front video and then switches to bottom view.

19th May 2015 — USB Ghost

Last night I tried to read something about USB devices and how they can be virtually unplugged. The situation was the same as during the contest — missing both /dev/ttyUSB0 and /dev/ttyUSB1. It turned out that I was actually lucky, because index 0 and 1 depends on the order how you plug-in the cables and motor controller was expected on /dev/ttyUSB1.

In particular I was looking for use cases of /sys/bus/usb/drivers/usb/bind and /sys/bus/usb/drivers/usb/unbind.

administratorcpr-sylv-01:~$ ls /dev/ttyU*
ls: cannot access /dev/ttyU*: No such file or directory

administratorcpr-sylv-01:~$ lsusb -t
6-1:1.0: No such file or directory
6-2:1.0: No such file or directory
/:  Bus 09.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 5000M
/:  Bus 08.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
/:  Bus 07.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 5000M
/:  Bus 06.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=vend., Driver=, 12M
    |__ Port 2: Dev 3, If 0, Class=vend., Driver=, 12M
/:  Bus 05.Port 1: Dev 1, Class=root_hub, Driver=ohci_hcd/2p, 12M
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=ohci_hcd/5p, 12M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=ohci_hcd/5p, 12M
    |__ Port 5: Dev 3, If 0, Class=vend., Driver=xpad, 12M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci_hcd/5p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci_hcd/5p, 480M

administratorcpr-sylv-01:~$ lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 006 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 007 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 008 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 009 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 003: ID 046d:c21f Logitech, Inc. F710 Wireless Gamepad [XInput Mode]
Bus 006 Device 002: ID 0403:6001 Future Technology Devices International, 
Ltd FT232 USB-Serial (UART) IC
Bus 006 Device 003: ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port

I also tried to complete the upgrade with -f option, but failed again on

grub-probe: error: cannot find a device for / (is /dev mounted?).
Installation finished. No error reported.
/usr/sbin/grub-probe: error: cannot find a device for / (is /dev mounted?).
dpkg: error processing grub-pc (–configure):
 subprocess installed post-installation script returned error exit status 1
Processing triggers for libc-bin …
ldconfig deferred processing now taking place
Processing triggers for initramfs-tools …
update-initramfs: Generating /boot/initrd.img-3.2.0-54-generic

I am mentioning this if it could be related to ghost behavior because after reboot I have seen the USB devices:

administratorcpr-sylv-01:~$ find /sys/bus/usb/devices/usb*/ -name dev
/sys/bus/usb/devices/usb1/dev
/sys/bus/usb/devices/usb2/dev
/sys/bus/usb/devices/usb3/dev
/sys/bus/usb/devices/usb3/3-5/dev
/sys/bus/usb/devices/usb3/3-5/3-5:1.0/input/input9/event9/dev
/sys/bus/usb/devices/usb3/3-5/3-5:1.0/input/input9/js0/dev
/sys/bus/usb/devices/usb4/dev
/sys/bus/usb/devices/usb5/dev
/sys/bus/usb/devices/usb6/dev
/sys/bus/usb/devices/usb6/6-1/dev
/sys/bus/usb/devices/usb6/6-1/6-1:1.0/ttyUSB0/tty/ttyUSB0/dev
/sys/bus/usb/devices/usb6/6-2/dev
/sys/bus/usb/devices/usb6/6-2/6-2:1.0/ttyUSB1/tty/ttyUSB1/dev
/sys/bus/usb/devices/usb7/dev
/sys/bus/usb/devices/usb8/dev
/sys/bus/usb/devices/usb9/dev

I do not know.

22th May 2015 — OS Upgrade

I decided to go ahead with upgrade of Husky OS. Martin from CPR confirmed that: Our Indigo release is available and stable. There was quite a bit of clean up and new features, so I would indeed suggest upgrading! Instructions to do so can be found here. Be sure to back up your system just in case! Then it was a kind of when if not now and who if not me question (there are three events at the beginning of next month and I need Husky working for the demo).

There were couple minor issues like I do not have a keyboard or monitor at home (we use only laptops and „smart” phones), so I borrowed one at work and used TV as monitor. Yes, Husky is now our second „dog” and lives in our living room. And yes, you can imagine how happy is my wife about it.

The second detail was USB disk. Based on the suggested link http://wiki.ros.org/Robots/Husky I downloaded relatively small (29MB) ISO image and used Universal-USB-Installer-1.9.5.9.exe to create bootable USB drive. That worked fine for other disk and other system, except that here I selected last option Try Unlisted Linux ISO. The format of USB disk for some unknown reason failed and maybe that was the detail causing problems later — Husky BIOS did not see it (I could mount it and use it with the system, but not from BIOS). So I used another USB disk, format it manually and then I could see two more options in boot list (F7 option). I did not write it down but one started from T and that worked while the other from U, which did not work.

There was another „stopper” before this step — it is highly recommended that you backup whole disk and the first step new installation will do is to wipe your disk! Do I really want to reinstall all the packages I installed over the year? Will it work in new system? It is hard to tell and I am sure I forgot some details like automatic ssh-tunnel for remote access …

The instructions were clear enough, I would maybe only stress a little bit that the internet is must have and you should have rather fast connection to download all the packages. Actually I had to fill only IP, mask, gateway and DNS server and the rest was automatic up to changing the computer name. There was one moment when I was particularly nervous (586/718) when for several minutes nothing was happening (based on „progress bar”)?! But now is the system replaced.

I tried ./rr.py, but it did not work, because Python wrapper (clearpath.horizon) is not installed. Then I tried my husky.py, which worked, but Husky did not move. I suppose that I did not configure sending necessary messages which was formerly done with ROS on boot up. It was 1am so I rather went to sleep and starting now at 5am … is it really so much fun?

As my grandfather would say: „it is done, just to finish it”.

27th May 2015 — ssh tunnel

This note is rather for myself if I ever have to reinstall Husky again. I had working ssh tunnel in start up script so whenever was Husky connected to the internet I was able to connect to it from dedicated server. It worked fine for me. A friend of mine did it for me and I did not know all the details. And after reinstall I lost that.

There were two issues: how to start it always on boot and where is home for root user (for .ssh configuration). The first was solved with /etc/rc.local file. It was necessary to add line before exit 0 and redirect input, output and error output, and run it with &.

The second was sorted with hint from askubuntu.com in particular line

sudo su - root

and then with echo $HOME I got searched /root. Yes, like total beginner. Now it establishes tunnel again and I am installing OpenCV (again). There is still unsolved issue with IMU, which requires calibration. This means turn in place which I would rather do outside …

p.s.

OpenCV 3.0.0-rc1 ready to be used

28th May 2015 — Compass calibration

It is still Wednesday evening, but … before I finish, it could be already Thursday. I was bit lazy to carry Husky outside so I decided to do the calibration at home at the end. This was after I read the source code of calibrate_compass where you can see what it will in reality do:

turn for 60 seconds
the speed is 0.2 (radians) which corresponds to 12 deg/sec
in total two slow complete spins

Result:

administratorcpr-sylv-01:~$ rosrun husky_bringup calibrate_compass
rospack: error while loading shared libraries: librospack.so: cannot open shared
 object file: No such file or directory
Traceback (most recent call last):
  File "/opt/ros/indigo/bin/rostopic", line 35, in 
    rostopic.rostopicmain()
  File "/opt/ros/indigo/lib/python2.7/dist-packages/rostopic/__init__.py", line 1753, 
in rostopicmain
    import rosbag
  File "/opt/ros/indigo/lib/python2.7/dist-packages/rosbag/__init__.py", line 33, in 
    from .bag import Bag, Compression, ROSBagException, ROSBagFormatException, 
ROSBagUnindexedException
  File "/opt/ros/indigo/lib/python2.7/dist-packages/rosbag/bag.py", line 65, in 
    import roslz4
  File "/opt/ros/indigo/lib/python2.7/dist-packages/roslz4/__init__.py", line 33, in 
    from ._roslz4 import *
ImportError: libroslz4.so: cannot open shared object file: No such file or directory
ROS appears not to be running. Please start ROS service:
sudo service ros start

Hmm, strange. I found roscd-and-roslaunch-problem that this was due to not properly set LD_LIBRARY_PATH. After setting it

export LD_LIBRARY_PATH=/opt/ros/indigo/lib/

it went through

administratorcpr-sylv-01:~$ rosrun husky_bringup calibrate_compass
Started rosbag record, duration 60 seconds, pid [2483]
Started motion commands, pid [2484]
[ INFO] [1432750402.337673856]: Subscribing to /tf
[ INFO] [1432750402.347888180]: Subscribing to /imu/data
[ INFO] [1432750402.356517429]: Subscribing to /imu/mag
[ INFO] [1432750402.372649791]: Recording to /tmp/calibrate_compass.oXJK/imu_record.bag.
Test underway.
Time remaining: 0
Shutting down motion command publisher.
Waiting for rosbag to shut down.
Computing magnetic calibration.
Calibration generated in /tmp/calibrate_compass.oXJK/mag_config.yaml.
Copy calibration to /opt/ros/indigo/etc/husky_bringup? [Y/n]
[sudo] password for administrator:
Restart ROS service to begin using saved calibration.

OK fine.

It turned out that the LD_LIBRARY_PATH was actually by default set:

administratorcpr-sylv-01:~/git/husky/ros$ env | grep LD
LD_LIBRARY_PATH=/opt/ros/indigo/lib:/opt/ros/indigo/lib/x86_64-linux-gnu
OLDPWD=/home/administrator

but the problem was with my favorite tool screen (I use it for the case the connection is broken).

On the other hand /imu/rpy still does not work (it is in rostopic list, but:

administratorcpr-sylv-01:~$ rostopic echo /imu/rpy
WARNING: topic [/imu/rpy] does not appear to be published yet

The same for /imu/temperature or /husky/data/encoders. The /imu/data works fine.

p.s. in meantime I started to work on alternative backup solution. I picked several years old board alix3d2 from „intelligent home” project, changed static IP to 192.168.1.12, set gateway to 192.168.1.1 and dns-namespaces to 8.8.8.8, plugged Ethernet cable into Husky switch, installed git and python-serial, downloaded husky.git, inserted USB for motion control and IMU, and was almost ready to go. Minimal dependences, much lower power consumption (I know for other projects we may need stronger PC). I had to change the code a little (diff) to request basic info from motion controller (originally pre-requested by ROS master), and that was it. Husky moved and logged all data.

29th May 2015 — imu.py

You'll learn to love ROS, just you wait ;) … this sentence is repeated in my mind again and again (the author is Martin from CPR). I do not know. Maybe if everything works, but it usually does not. It is not ROS failure, it is just complex system under development. The problem can be in the cable, bad configuration, burned capacitor or who knows? For me it is still extra level which sometimes hides the problems and makes the whole thing even more complex.

Friends at Robotem Rovne 2015 said „we use ROS, we have to, but we do not like it”. They told me stories about colliding libraries and dedicated systems only for ROS. On the other hand roboauto is using ROS for their autonomous car and also commercial mobile platforms mir-robotics is using ROS. Also apple harvesting robot was using ROS. I do not know.

At the moment I do not have working ROS system. I believe that it is just some simple configuration somewhere, but somehow I prefer to learn and understand the foundations on which are the ROS nodes based. I had problems with IMU year ago, so this time I started from scratch (see imu.py today's history). It looks OK, it is well documented and I am starting to be confident to build demo based on it for next Tuesday (http://dnipola.sk/). At the moment imu.py is parsing only temperature, but I also see what messages it is sending:

UM6_GYRO_RAW_XY = 0x56
UM6_GYRO_PROC_XY = 0x5e
UM6_MAG_PROC_XY = 0x60
UM6_EULER_PHI_THETA = 0x62
UM6_TEMPERATURE = 0x76

And the driver expects Euler angles and quaternions … so maybe the calibration actually broke the IMU node??? But it was not sending the /imu/rpy before, so probably not. Note, that now I am using another board, not the mini PC with ROS, so it could be interesting if the ROS node is changing UM6 configuration?? I am just running out of time, so „another day”.

3rd June 2015 — dnipola.sk/Encoders

It is early in the morning and I am preparing Husky for today's show. Yesterday it went fine — I was using only rr.py code from Robotem rovně and I was flying with the drone above the heliport sign. So far so good. Note, that I installed clearpath_py_r920.tar.gz on both computer boards in order to run rr.py.

I made two observations:

the battery was really „drained” by the hungry computer — using the alternative board solved the problem. Husky was running all day on one battery charge … yes it was mostly off, or not moving but still 100 times better than when I was watching power bar going to zero before. Also there were no overheating issues even during the hot day.
the communication once a while fails and it looks like the Husky temporarily stops (very short „blink” of red light)

Today I planned to repeat the yesterday show, maybe with some extra maneuvers, so that people will see the drone also turning. The simplest solution was to add extra bits of code into husky.py (diff) and I was surprised (again) to see raw data from encoders with status of emergency STOP button:

True (2, 0, 0, 0, 0)
True (2, 0, 0, 0, 0)
False (2, 0, 0, 0, 0)
False (2, 0, 0, 0, 0)
False (2, 1, 0, 25, 20)
False (2, 3, 3, 34, 32)
False (2, 8, 7, 28, 27)
…
False (2, 218, 212, 11, 6)
False (2, 218, 212, -23, -17)
True (2, 215, 210, -32, -24)
True (2, 214, 208, -29, -26)
True (2, 214, 207, -29, -24)
True (2, 214, 207, -29, -24)
…
True (2, 214, 207, -29, -24)
True (2, 214, 207, -29, -24)
False (2, 214, 207, -29, -24)
False (2, 214, 0, 0, 0)
False (2, 216, 2, 35, 49)

So shortly after eStop was released the second encoder was reset to zero while the first encoder continued from origin value?! I am still convinced that something like this can case the „jumps” I was observing when using ROS couple weeks ago … sigh.

Just remark, I would not use eStop for starting/pausing my program but it is the only button available on Husky :-(.

contact form

Katarina

2015-01-28T00:00:00Z

This is a blog about my experience with Parrot Bebop drone. She received "working name" Katarina … I expected that it will be roar and originally wanted to name it Katrina, but I changed my mind as there was too much destruction behind that name.

At the moment I am not sure if I will write this blog in Czech or English — we will see if the world is more interested then Czech community … at the moment it will be mix of both.

The drone is loan from Czech Parrot distributor (last time it was minidrone Jessica). Thanks

Box

Packed Katarina

Size

Content

Blog

31st January 2015 — First talk to "dragon"

I would like to write some notes in parallel to github development, similar to other projects. It helps me to remember some details and it could better explain my steps for random reader/developer.

Today I unpacked Bebop and powered it up. It was quite noisy even I did not take off … there is a fan for the processor probably (?). My first task was to take a picture but now I am happy that the drone started to talk to me .

I would recommend document Parrot Bebop Drone Hacking. But the hint, how to establish communication with the drone, is from here. It is necessary to do „discovery step” before you can use ports c2d_port and d2c_port. TCP is used on 192.168.42.1, port 44444 and you have to first send JSON to the drone with controller_type, controller_name and d2c_port. See diff for details.

The Python scripts sent:

{"controller_type":"computer", "controller_name":"katarina", "d2c_port":"43210"}

and received:

{ "status": 0, "c2d_port": 54321, "arstream_fragment_size": 1000, 
"arstream_fragment_maximum_number": 128, "arstream_max_ack_interval": 0, 
"c2d_update_port": 51, "c2d_user_port": 21 }

To be continued .

2nd February 2015 — Parsing Navdata

At first I would like to thank Darryl for his supportive mail . It pushed me to get some data from d2c_port. I started with copy and paste from Heidi code (UDP sockets), so I call them again NAVDATA_PORT and COMMAND_PORT.

The code was simple:

robot = Bebop() 
for i in xrange(100): 
    robot.update( cmd=None )

and because it already generated logs with date/time I have test for 1, 10 an 100 messages (see related diff).

The structure looks similar to Jessica (Rolling Spider). Code 2, then type of queue (?), increasing index and length of the whole data packet. There is now navdata.py for separate parsing and if you run it on logged data you will see something like:

m:\git\ARDroneSDK3\katarina>navdata.py navdata_150201_211239.bin
127 1 size 35
127 2 size 23
127 3 size 23
127 4 size 19
127 5 size 23
127 6 size 23
127 7 size 19
127 8 size 23
127 9 size 23
127 10 size 19
0 1 size 15
127 11 size 23
127 12 size 23
127 13 size 19
127 14 size 23
127 15 size 23
127 16 size 19
…

The counters are separate and so far it looks like that the 0-queue contains only 15bytes long message.

This is the content:

02 00 01 0F 00 00 00 5D 01 00 00 99 56 57 0C
02 00 02 0F 00 00 00 5E 01 00 00 87 C8 93 0D
02 00 03 0F 00 00 00 5F 01 00 00 A7 9E A7 0D
02 00 04 0F 00 00 00 60 01 00 00 CF 39 B7 0D
02 00 05 0F 00 00 00 61 01 00 00 0D EB F7 0E
02 00 06 0F 00 00 00 62 01 00 00 A2 85 0E 0F

So again some counter and checksum or time??

In 127 or 0xFE "queue" the 19bytes long messages looks the same all the time (except increasing counter):

02 7F 04 13 00 00 00 01 04 08 00 00 00 00 00 00 00 00 00

23 bytes message are two types:

02 7F 02 17 00 00 00 01 04 06 00 E8 6D E6 3C F5 7D 20 BC AA 4D 85 3B
02 7F 03 17 00 00 00 01 04 05 00 00 00 00 00 00 00 00 00 00 00 00 00
02 7F 05 17 00 00 00 01 04 06 00 5B 48 E8 3C 08 D0 1E BC 28 F0 84 3B
02 7F 06 17 00 00 00 01 04 05 00 00 00 00 00 00 00 00 00 00 00 00 00

And finally 35 bytes message is so far still the same.

02 7F 01 23 00 00 00 01 04 04 00 00 00 00 00 00 40 7F 
   40 00 00 00 00 00 40 7F 40 00 00 00 00 00 40 7F 40
02 7F 11 23 00 00 00 01 04 04 00 00 00 00 00 00 40 7F 
   40 00 00 00 00 00 40 7F 40 00 00 00 00 00 40 7F 40

Well, it may take a while to find corresponding file in ARDroneSDK3 …

3rd February 2015 — generateLibARCommands.py

You would not find useful file, you have to generate it first! In particular you will need libARCommands and probably also main ARSDKBuildUtils.

Then you need to locate generateLibARCommands.py and run it with parameter -projects ARDrone3 (you will not find code name "Bebop").

m:\git\ARDroneSDK3\libARCommands\Xml>generateLibARCommands.py -projects ARDrone3

Then you can find desired files: ARCOMMANDS_Ids.h, ARCOMMANDS_Filter.c and ARCOMMANDS_Decoder.c. Interesting lines from ARCOMMANDS_Filter.c are:

commandProject = ARCOMMANDS_ReadWrite_Read8FromBuffer (…);
commandClass = ARCOMMANDS_ReadWrite_Read8FromBuffer (…);
commandId = ARCOMMANDS_ReadWrite_Read16FromBuffer (…);

So if you look for byte commandProject = 1 (ARCOMMANDS_ID_PROJECT_ARDRONE3), you will find combinations for example 01 04 04 00 in 35bytes messages in previous post. commandClass = 4 (ARCOMMANDS_ID_ARDRONE3_CLASS_PILOTINGSTATE) and commandId = 4 (ARCOMMANDS_ID_ARDRONE3_PILOTINGSTATE_CMD_POSITIONCHANGED). Now you can lookup decoding routing in ARCOMMANDS_Decoder.c:

_latitude = ARCOMMANDS_ReadWrite_ReadDoubleFromBuffer (…);
_longitude = ARCOMMANDS_ReadWrite_ReadDoubleFromBuffer (…);
_altitude = ARCOMMANDS_ReadWrite_ReadDoubleFromBuffer (…);

All three 8bytes values are 00 00 00 00 00 40 7F 40.

ARCOMMANDS_ID_ARDRONE3_PILOTINGSTATE_CMD_SPEEDCHANGED = 5 … float _speedX, _speedY, _speedZ;
ARCOMMANDS_ID_ARDRONE3_PILOTINGSTATE_CMD_ATTITUDECHANGED = 6 … float _roll, _pitch, _yaw;
ARCOMMANDS_ID_ARDRONE3_PILOTINGSTATE_CMD_ALTITUDECHANGED = 8 … double _altitude;

That could be enough to get started . See related diff.

p.s. I expected copy and paste error, but it is different "ATTITUDE" vs. "ALTITUDE"

4th February 2015 — Commands required

I asked Darryl to repeat my first step and in meantime I also did it myself this morning. And it did not work :-(. At least 100 messages did not go through. When I set the limit to 10 or even 50 it was OK. So there is some time limit when you have to send some command to the drone.

I did not mention this, but just for reference I am using Windows 7 and Python 2.7. On the other hand it should work also on other OSs so far (at the moment there is no dependency, but it will probably soon change and OpenCV2 and NumPy libraries will be required).

So how to send the command and which command would not be too dangerous? It is surely not a good idea to send takeoff first . There are commands like SetHome and ResetHome which should generate some drone response and without flying they should not be harmful.

The encoding scheme is similar to decoding:

ARCOMMANDS_ReadWrite_AddU8ToBuffer(…ARCOMMANDS_ID_PROJECT_ARDRONE3…);
ARCOMMANDS_ReadWrite_AddU8ToBuffer(…ARCOMMANDS_ID_ARDRONE3_CLASS_GPSSETTINGS…);
ARCOMMANDS_ReadWrite_AddU16ToBuffer(…ARCOMMANDS_ID_ARDRONE3_GPSSETTINGS_CMD_RESETHOME…);

and

ARCOMMANDS_ReadWrite_AddU8ToBuffer(…ARCOMMANDS_ID_PROJECT_ARDRONE3…);
ARCOMMANDS_ReadWrite_AddU8ToBuffer(…ARCOMMANDS_ID_ARDRONE3_CLASS_GPSSETTINGS…);
ARCOMMANDS_ReadWrite_AddU16ToBuffer(…ARCOMMANDS_ID_ARDRONE3_GPSSETTINGS_CMD_SETHOME…);
ARCOMMANDS_ReadWrite_AddDoubleToBuffer(…_latitude…);
ARCOMMANDS_ReadWrite_AddDoubleToBuffer(…_longitude…);
ARCOMMANDS_ReadWrite_AddDoubleToBuffer(…_altitude…);

The question is what is necessary to send before these bytes are encoded??

I will at least answer one Darryl's question: How could you tell about counter, checksum, and time?? Well, I was probably wrong with checksum, and time or better timestamp I guessed as the 4 byte number was increasing (and I am maybe wrong). But I am pretty sure about the counter. Parrot used that in ARDrone2 (absolute increasing uint32) and also for Rolling Spider (uint8). The packets can be lost in UDP communication and one way how to recognize this fact is to index them. The packets also do not have to come in the original order although it is probably not the case with direct WiFi connection.

5th February 2015 — libARNetworkAL

Lesson learned: if you want to play with ARDrone3 download ALL 14 repositories!!! Do not be lazy like me! Otherwise you may omit for example libARNetworkAL. It is the place where you will find this structure:

typedef struct
{
    uint8_t type;     /* frame type eARNETWORK_FRAME_TYPE */
    uint8_t id;       /* identifier of the buffer sending the frame */
    uint8_t seq;      /* sequence number of the frame */
    uint32_t size;    /* size of the frame */
    uint8_t *dataPtr; /* pointer on the data of the frame */
}

Not interested? Well, review the collected data now . 02 is type and it corresponds to ARNETWORKAL_FRAME_TYPE_DATA — Data type. Main type for data that does not require an acknowledge.

The next item was 00 or 7F and it is id. The seq was guessed correctly but size is not single byte, but whole uint32, which explains the three zeros. And the rest we were already able to decode (see diff).

So now, what to do with the problem from yesterday? How to encode whole command packet? Based on ARNETWORKAL_WifiNetwork.c and function ARNETWORKAL_WifiNetwork_PushFrame it should be the same frame as we just decoded. I would guess that like for Jessica the type=2 but I am not sure about id. According to ARNETWORK_IOBufferParam.h in libARNetwork ID — Identifier used to find the IOBuffer in a list - Valid range : 10-127.

Here is my first attempt to send command — unsuccessful :-(. But I received new messages . In particular

04 7E 01 23 00 00 00 01 18 00 00 00 00 00 00 00 40 7F C0 
         00 00 00 00 00 40 7F C0 00 00 00 00 00 40 7F C0

where 04 is message with required acknowledgement (ARNETWORKAL_FRAME_TYPE_DATA_WITH_ACK), 01 18 00 00 is ARCOMMANDS_ID_PROJECT_ARDRONE3 = 1, ARCOMMANDS_ID_ARDRONE3_CLASS_GPSSETTINGSSTATE = 24, ARCOMMANDS_ID_ARDRONE3_GPSSETTINGSSTATE_CMD_HOMECHANGED = 0 … so maybe I was sucessful at the end! . This worked for id=10, but did not work for 0x7F and 0x40.

Confirmed . So one more small diff. Also for frameId=10 I received all 10+100 messages, while for other two "random" channels it stopped before 100.

OK, so the next step will be confirmation of the received message (type 4).

15th February 2015 — PING and PONG

Well, nothing really exiting yet . I tried to confirm (acknowledge) message of GPS home changed, and I did not succeed. But in that particular piece of code (see original ARNETWORK_Receiver.c:191) there is an extra switch for handling frame.id ARNETWORK_MANAGER_INTERNAL_BUFFER_ID_PONG. If you look this ID up you will find that it is 0. And what is it?

It looks like that these are the 15bytes long messages starting with 02 00 XX 0F prefix. The payload is struct timespec, which should be 8 bytes (time_t and long), probably both 4 bytes (seconds, nanoseconds).

Time 176.160592961
Time 177.181624819
Time 178.202618948
Time 179.362471135
Time 180.38329366

So it looks like PING every second, and because it is not PONGed the connection is lost after a while (I guess).

There was a small surprise regarding UDP packets. One packet can contain several messages and thus simple logging (see diff) is not sufficient. So far I have seen only combination of standard messages with these PING messages, but who knows …

Update:

the UDP packets are now split for reliable record and replay + asserts On/Off (diff)
I got ping-pong working and tested it on 1000 messages (diff)
thanks to Jumping Sumo sample I know how the configuration looks like … frame.id = 10/127 for normal messages, 11/126 for acknowledged
I received new message 04 7E 02 0C 00 00 00 00 05 01 00 38 which should mean ARCOMMANDS_ID_PROJECT_COMMON = 0, ARCOMMANDS_ID_COMMON_CLASS_COMMONSTATE = 5, ARCOMMANDS_ID_COMMON_COMMONSTATE_CMD_BATTERYSTATECHANGED = 1, value 0x38=56% … so it is probably OK, that I did not receive this always

16th February 2015 — ACK of ACK?

There was a big mistake in the code yesterday — yes, I wrote that code. It is necessary to pack payload for normal command but special cases like ACK or PING are already prepared to be sent as-they-are (see fix).

This means that ping-pong was probably not working, only the communication kept the drone busy. After the change I received strange 8 bytes messages. They had frameType 0x1 (ARNETWORKAL_FRAME_TYPE_ACK) and they were coming with frameId 0x8B?! In HEX code it is more visible. Now I would guess that it was acknowledge of my (wrongly composed) acknowledge of original home changed or battery status changed from frameId 11 = 0xB.

I am convinced now that frameId 10 is for sending standard messages without acknowledgement and 11 for any (?) messages for which I would like to get acknowledgement. I did not succeed to send ACK probably, because the message from the drone was still repeated. The plan is to try (again) frameType 0x1 (ARNETWORKAL_FRAME_TYPE_ACK — Acknowledgment type. Internal use only … but I am probably working on "internal" now), and frameId = 0x7E + 0x80 … I wonder what is the chance of success :-(.

p.s. yesterday I have seen the drone die on low battery again. The last message was something like Battery 32%, so the main computer eats a lot or it has set auto-shutdown threshold relatively high.

p.s.2 it looks like it worked (diff) …

17th February 2015 — Low latency video stream

It is time for the next step. I found ARCOMMANDS_ID_ARDRONE3_MEDIASTREAMING_CMD_VIDEOENABLE in the list of IDs and tried to call it (see diff). There were many new messages almost 1kB long and they looked like:

03 7D 7D F4 03 00 00 07 00 01 00 1F 00 00 00 01 27 42 E0 28 …
03 7D 7E F4 03 00 00 07 00 01 01 1F DE 11 5A 4E 70 5E BE F6 …
03 7D 7F F4 03 00 00 07 00 01 02 1F 96 C4 44 3E A3 7F 0E 0D …
03 7D 80 F4 03 00 00 07 00 01 03 1F D1 8B 07 A2 57 2A 86 C7 …
…

At first I was happy to see 00 00 00 01, which is used as start tag in H.264 video codec. I cut that part and tried to replay it with old Heidi code. I saw image patches of reality, but also many errors.

Later I recorded longer communication (instead of 100 messages it was 1000 and 5000) and I was surprised that not all packets arrived and that they are repeating. Why? The answer is simple: they have to be acknowledged.

Note 03 at the beginning — this is a new type of frames (ARNETWORKAL_FRAME_TYPE_DATA_LOW_LATENCY). Also note, that there is a new frameId FD and according to Jumping Sumo Receive Stream sample could be the acknowledge frameId equal to JS_NET_CD_VIDEO_ACK_ID = 13.

Both I/O structures look reasonable (see libARStream/ARSTREAM_NetworkHeaders.h):

typedef struct {
    uint16_t frameNumber; /** id of the current frame */
    uint8_t frameFlags; /** Infos on the current frame */
    uint8_t fragmentNumber; /** Index of the current fragment in current frame */
    uint8_t fragmentsPerFrame; /** Number of fragments in current frame */
} __attribute__ ((packed)) ARSTREAM_NetworkHeaders_DataHeader_t;

So in my example 07 00 01 00 1F is frameNumber = 7, frameFlags is so far always 1 (FLUSH FRAME), fragmentNumber is increasing and 1F is fragmentNumber.

Now it is time to implement video packet acknowledgement:

typedef struct {
    uint16_t frameNumber; /** id of the current frame */
    uint64_t highPacketsAck; /** Upper 64 packets bitfield */
    uint64_t lowPacketsAck; /** Lower 64 packets bitfield */
} __attribute__ ((packed)) ARSTREAM_NetworkHeaders_AckPacket_t;

i.e. there are up to 128 frame fragments which have its own confirmation bit.

18th February 2015 — video.py

It looks like the video packet confirmation works (see code diff). Note, that the frame fragments can be in random order and there could be also some duplicities. Because I wanted to see the video and I was not patient enough, I wrote simple brute force utility which sorts the fragments in the memory and creates video file (see video.py). And it works.

I am using code from Heidi to replay the video file (rr_drone.py or directly airrace.py). But because the file does not contain PaVE headers any more, you can directly load it with OpenCV:

import cv2
cap = cv2.VideoCapture( "video.bin" )
ret, frame = cap.read()
cv2.imwrite( "first-video.jpg", frame )

and this is my first result:

first successfully transfered video frame

I know that the picture of the side view of our fridge is not very exciting.

If you want just to see the frame(s), use:

cv2.imshow('image', frame)
cv2.waitKey(0)

with wait in milliseconds or 0 for pause until user press any key. And finally if you want your code to be as it should be clean it all at the end:

cap.release()
cv2.destroyAllWindows()

Yesterday I finally took off. It was only in Free Flight 3 application but still it was exciting. The bad news is that Bebop has the same problem as Rolling Spider — you cannot land until you complete takeoff sequence :-(. That means troubles and I hope Parrot will soon change this in the future firmware.

The second observation was related to image stabilization — I did not notice it before, or there is some setting to turn it on/off, but if you tilt your drone the image does not tilt . The same if you point up and down it remains stable. The application asked for magnetometer calibration before (which I did not because I was afraid that it will take off), and yesterday we calibrated it. So maybe that was why it is now working?

What do to next? I plan to try move image ROI (region of interest), switch to HD video resolution (from the picture you can see that it was 640x368), take 14Mb resolution picture … and probably finally takeoff, move and land autonomously .

20th February 2015 — Camera Tilt/Pan and Emergency

Yesterday I experimented with camera tilt and pan (diff). It was fun, because it worked immediately . Yes, except detail, that the byte for setting camera angle is signed, so packing once failed, but the fix was trivial.

I implemented also utility play.py — basically a short code for playing video as I was describing it in the previous report. Now, as soon as you finish your test, you can replay your video directly from navdata log file.

This morning I tried first takeoff(). It is always exciting, kind of adrenalin fun, because you do not know what will happen. I was holding Katarine tight, and called robot.takeoff() followed by robot.emergency() from Python script, i.e. cut motors as soon as you start them. What would you expect to happen? The propellers were still turning after the script termination!

You can turn Bebop upside down and that stops the motors, but still … it was surprise even I was ready for everything. Here is the code plus extra lines for parsing new messages: calibration info, flight number info and piloting state.

I suppose that there will be the same problem like with Rolling Spider, i.e. I will have to wait until take off sequence is completed before I can land :-(. BTW this is one of the reasons I am bit afraid to takeoff in this narrow space.

21st February 2015 — First Crash

„I told you there will be a problem!” … and yes, there was. I partially moved furniture in the living room to have enough space for the first takeoff test (code diff). The first test was over quickly: battery low, battery critical and stop, so the propellers hardly twisted.

I swapped batteries (Bebop has second battery set in the standard package, which is very nice), took off and during hover the drone slided a little bit to the left, hit the wear drier and landed. The video is not very exciting, so I picked at least one video frame recorded by the drone, when it is falling/landing and it sees its own protective hull:

Falling down

Later that day I did also outdoor test with Free Flight 3 application. And that looked even more scary. There was a light breeze and at one moment the drone was completely uncontrollable (WiFi connection lost?). I wander how the black box works and if there are any data stored? I am also curious if by default was the Flight No XXX recorded?

Sigh. Nothing. Based on Bebop hacking page I would have to have enabled Black Box in /etc/debug.conf. During the tests at underground garage we probably turned off automatic video recording, so there is nothing from yesterday. Sigh. On the other hand the quality of recorded garage video is great (full HD, stable image).

22nd February 2015 — ManualControlException and FlatTrim

Katarina is finally slowly learning the same commands as Heidi. Today I added ManualControlException, something like red Emergency STOP button on every robot. For the flying drone this means land as soon as you hit any key on the keyboard. The bad news is that there is no platform independent kbhit so at the moment it works only under Windows. There was implementation with pygame on Linux version (Isabelle), but I did not test it yet and it is necessary to create some window and initialize pygame library. An alternative is Joris's Linux version — TODO (let me know if you would like to use Katarina's code on Linux or Mac). Here is the slightly bigger diff. Note, that apyros/sourcelogger.py is copy from Heidi code.

Any exciting moments? … well at least two. First of all the drone was always moving to the left as soon as it switched to hover mode. So yesterday it was not necessary the breeze, but probably missing flat trim. Fixed. Now it stays in place.

The second funny moment was that I forgot land() command in normal run. When I hit the emergency button everything was fine, but then I let that be, actually I hit the button outside the try..except block and the drone was still flying and flying. New lesson is that you can grab the Bebop with the same trick as ARDrone2 — one hand from the top, so you do not disturb down pointing camera and sonar, and then the second hand from the bottom, and twist up-side-down.

The last step I did today was some cleaning (see diff), so now command are in separate file, trim() is finished as soon as drone confirms it and takeoff() and land() is terminated by flying state change.

What next? Redirect incoming video frames to extra working thread and look for two-colors cap:

Sample image with the Parrot Cap

23rd February 2015 — Fandorama

I will switch this blog to Czech language for a while. I did not expect it, but at the end the fandorama rising project was successful

There will be surely updates on github, and I will probably write major changes/discoveries in both languages, but … let me know if you find this article/blog interesting, and if I should explain or translate some parts.

20th March 2015 — Status quo

It is hard to believe that it is almost a month since I switched to Czech „blogging”. There is some progress but it is still far from perfect . In particular I am fighting now with streamed video which sometimes stops and I have to power down the drone to restart it. No idea why.

Screenshot with cap color detection

On the other hand on-board video recording works fine, so I have „internal” reference, but that is useless for autonomous navigation. It is somehow related to flying … maybe I have to send ping messages myself to keep the drone going??

I did some experiments in our garden and it turned out that the grass is not well suited for landing (it was good in one crash case, but now I mean „standard successful automatic landing”). That was old task for ARDrone2 when the detection was done by Parrot's code. This is no longer true, so I have to do it myself:

Example of wrong detection

There is new folder behaviors, where you can find experimental navigation to box navbox.py. Image multiprocessing is prepared but not fully integrated yet. It should be fun as soon as the video stream is reliable.

What else? I did some experiments with navdata and black box according to the forum thread Hacking the Bebop. Both collects data on-board, including sonar and EKF estimates, but it is not streamed over network :-( … see Nicolas note on GitHub.

There is new Parro Bebop Protocol document. The motivation to write it down was from British forum, but no idea, if it was really useful for anybody …

Yeah and some fun from American forum … a fracking genius … can read hex like Neo reads the Matrix! … ha ha ha . No, not really. But speaking of HEX … I did some (more) experiments with H.264 codec decoding for ARDrone3 (in Czech H264 Drone Vision). And surprise, surprise … the codec is different. While ARDrone2 was sending only 16x16 macroblocks, you get finer compression on Bebop with sub-blocks down to 4x4. This means that the H264 navigation code needs some work to be done to port it on 3rd generation of Parrot's drones.

22nd March 2015 — PCMD @40Hz

This one is worth mentioning immediately. Now I know why the video streaming fails once a time! The reason is timing of PCMD messages … but let's start from the beginning.

On Saturday I finally created example when the video streaming fails without need of flying (see diff). Later I wrote even simpler example (diff), i.e. as soon as I sent the first PCMD the video stopped and was not restored until next power down and power up.

So what to do? I am glad I wrote to ARDroneSKD3/libARCommands/issues. And surprise surprise I got answer today!

When using the PCMDs, you should make sure that your send them with a fixed frequency (we use 40Hz / 25ms in FreeFlight), even if you're sending the same command every 25ms.

A change in the PCMD frequency is considered by the Bebop to be a "bad wifi" indicator. In this case, the bebop will decrease the video bandwidth (up to a "no video" condition if the indicator is really bad) in order to save the bandwidth for the piloting commands. When the PCMD frequency become stable again, the video bandwidth will rise again.

This explains a lot - in particular why it sometimes worked . If it was fixed loop like flying to requested altitude it was OK, but as soon as I switched to another task the timing was broken.

Here is small hack confirming Nicolas answer … it works, even for the second run. But … yes, there is still but, only during the times when the Tread is on sending PCMD at 40Hz, so I am asking now if it is necessary to send PCMD all the time, or if there is a way how to say „I do not want to send more PCMD!”.

8th April 2015 — Terminated.

Well, I know that The Day will come, but still it strike me out unprepared . The distributor asked me to return the drone because of some presentation. OK, what can I do. Katarina is not mine. I could buy one, but do I want to??

Interestingly enough it was after the Eastern weekend when I decided that if I want to move forward I will have to write the firmware myself or use/learn from others. Yesterday I downloaded ardupilot-mpng and paparazzi repositories … and then I received that mail.

Somehow I lost the motivation to practise landing yesterday and do the cap tracking with multiprocessing today. I deleted videos and thumbnails from the drone and I will return it in two hours … Goodbye Katarina!

14th April 2015 — Remote distributed debugging

Thanks Charles and others for mails, queries and issues on GitHub . I am not giving up yet . I do not have Bebop now, but that will probably change in May 2015. So now I am remotely debugging like with dropping incomplete video frames (diff). This one I did from old log files, but I can repeat the problems even from your new log files . Now I am waiting for Charles's confirmation that fix worked also for him.

At the moment I am studying paparazzi code for ARDrone2, but note, that there is also Bebop board code. The goal is to stream missing navdata like raw sonar readings … and maybe more. We will see.

30th April 2015 — Paparazzi?!

I had to smile yesterday, when I received notification from GitHub (regarding Mavlink seems not to work issue): If you really need to do it before the next firmware release (in the coming weeks), you can maybe look at Paparazzi. I can't provide support on it, but I think they support the bebop. It is not very easy, so maybe a small wait is better

It was hard to believe that this is the Parrot answer, but yes, it was. So I am probably on the right track learning Paparazzi for ARDrone2 at the moment, and I can confirm that reading navdata on Heidi onboard works.

Regarding Katarina I merged DEVELOP back to master — the issue with navigation Home is not solved (see Issue #7) and it is probably related to minimal height above the ground. Aldo did some experiments at 10 meters with string, but what he would need is rather MavLink. If nothing else then at least battery status and include paths in Linux are fixed and that was the main reason why I merged it.

contact form

AppleBot

2014-12-10T00:00:00Z

Content

Source code

https://github.com/robotika/applebot

Blog

3rd December 2014 — Apple 3D scan

Yesterday we did the first experiment with apple recognition at CZU. We used already existing tools:

TiM55x laser scanner
effector for linear motion (developed for some soil analysis)
laser.py from Eduro project

The sensor TiM55x we obtained thanks to SICK Robot Day 2014 participation, but we never throughly test it. It should be similar to older versions, but … there were two details:

the IP was just crazy: 169.254.225.156 (we had to use old notebook with SICK software to find this out. Once this was set correctly sensor responded also to ping commands)
if you send authorization command, used for LMS100, you got only old scan

The simple scanning script just collects data as fast as possible (over Ethernet) and stores them in date-time-filename generated log file(s).

The experiment was a with growing citrus tree with attached apples like for Christmas . And the results?

First of all TiM 55x has coarse resolution than former (and bigger) LMS 100 — 1 degree instead of 0.5 degree. 350 readings corresponded to 0.777m and it took 32.28 seconds. I.e. approximately 10Hz and 0.024m/s motion. So in X we have resolution 2.4mm and Y depends on apple size and distance from the scanner.

Would you like to see the results? I would recommend to download CloudCompare for 3D point clouds visualisation. It is necessary to convert laser scanner data into point cloud and you can use log2pts.py script. Here is also at least one example scan_141202_183934.zip. Note, that you have to unzip it first.

And here are the first views in Cloud Compare:

3D scan

pot detail

Small disappointment? Well, I have to admit, that I have not recognize it too at first sight. It is necessary to rotate it a little, you should easily recognize the railing and the three walls are ground, wall and the ceiling.

People probably still prefer camera images, right .

reality

4th December 2014 — Back to 2D

Today I will start with a reference to an article: Juan Feng, Gang Liu, Shengwei Wang, Lihua Zeng, Wen Ren, A Novel 3D Laser Vision System for Robotic Apple Harvesting, 2012 ASABE Annual International Meeting (direct link). They did something very similar to what I wanted to try — 3D laser scan of the tree and then search for the apples.

First of all here is the log2pgm.py utility so you can see scanned distance data as 2D image. If you run it on the data I posted yesterday you will get:

distance image with two apples

So it is maybe better to view 3D data as 2D image — it is in reality projection of the 3D world on a cylinder, so we have all necessary info there. Note, that I used fixed scaling of distances to grayscale. This way I will not loose distance information (when compared to the referenced article). I expect that scanned tree is not further away than 1 meter (otherwise it would require slightly different scale).

Do you see the two apples on the citrus tree? I do . The darker one is closer so it is larger. The image is flipped in Y-coordinate, as we scanned this particular one from right to the left, but it is just a detail for the moment.

So what next? Basically the same as we planned to do in 3D: we expect that the apple has some size limits. These corresponds to some bounding box in 3D. In grayscale image this means thresholding with narrow window and then searching for objects of given size.

There is another rule we can also add: the apple should not have holes, i.e. occluded parts have to be closer/darker otherwise detected object will be rejected.

5th December 2014 — cv2.MSER

I wanted to get Version0 running this morning, but I was a bit naive. At least I prepared necessary tools for 3D image analysis.

The first idea was to use threshold with lower and upper limits. You can play with the threshold yourself: all you need is output from log2pgm.py and cv2-bits/threshold.py. You can track both apples with minimum distance in the center and complete shape in 10 steps corresponding to 5cm.

Then I was thinking that even cv2.MSER will extract easier what we need (see code diff). MSER worked fine in AirRace rectangle detection, so maybe apple object are also well separable? Some are and some are not, see the temporary output:

output of MSER with restricted BBox size

You can see that darker left apple was not classified as candidate. It is clearly the nearby branches spreading the blob. On the other hand even the ligher (further away) apple was not obvious. I did not measure them, my fault, so I expected size like 10cm and this seems to be 6cm?? Here is the console input/output if you would like to repeat my steps:

m:\git\applebot>finder.py 0.06 logs\scan_141202_183934.txt
(380, 274) uint8
0.0696 (15, 138) (44, 149)
0.0528 (0, 132) (22, 141)
0.0504 (125, 138) (146, 148)
0.0648 (124, 137) (151, 153)
0.0672 (123, 137) (151, 156)
0.0648 (208, 116) (235, 139)
0.06 (171, 136) (196, 146)
0.0576 (75, 149) (99, 159)
None

I should also correct my statement about count of „apple points” as a function of distance to scanning device. In X coordinate the size (bounding box in pixels) should be the same. It is necessary to take into account that due to shaking the readings corresponding of nearby object will be much more reliable, but still, the size will be plus minus the same.

The difference is in Y coordinate is on the other hand significant. The resolution if given by scanner resolution (1 degree), so an apple in 50cm distance will have double the number of points when compared to an apple in 1 meter.

What next? I is necessary to filter out obviously wrong detection rectangles:

find median distance value
verify aspect ratio of detected rectangle (closer apple should be thin while further thick)
verify compactness of the surface (no holes)

I would guess that these simple steps will already give the apple but let's wait for real verification.

And what after that? Well, now we have the whole 3D scan in powerful NumPy array so once we have potential object we can return back to high resolution (laser readings are in millimeters) and play for example with surface normals. I would be quite curious if you can really reliably extract them and then try 3D object matching like in Model Globally, Match Locally: Efficient and Robust 3D Object Recognition paper.

6th December 2014 — Three apples?!

I must admit that the experiment last Tuesday was a bit spontaneous and now I have the impression that the very last 3D scan (the one I uploaded data) is actually with 3 apples! That some joker placed extra apple on the box (the next time we will automatically record it with Eduro IP camera to have reference pictures with timestamps):

candidates detected by MSER

In mean time I received dimensions of scanned apples:

smaller one is 7cm in diameter and 5.5cm high
yellow apple is 7.5cm in diameter and 7cm high

I should also mention a bugfix of numpy array conversion. I am learning it and I must admit it is very powerful library. The bug was in my conversion from int32 to uint8 and that was the source of horizontal lines. The corrected code looks like this:

tmp = np.array( scans ) / 5
mask = tmp > 255
tmp[mask] = 255

So you can define Boolean mask and then modify all elements where the mask is True with single operation (see hint source at stackoverflow.com).

Here is code used for above picture. The parameters for MSER are slightly modified (threshold step 8 instead of 1) and there is a new function isItApple() for verification of the patch (in this context a submatrix of original 3D scan/array). Also inside (half the size in X and Y so 1/4th in area) elements are dumped:

m:\git\applebot>finder.py 0.065 logs\scan_141202_183934.txt
(380, 274) uint8
0.0672 (123, 137) (151, 156)
205
[ [102 102 103 255 101 101 102 255 255  50  79 107 103 101]
 [126 126 125 126 125 125 126 125 125 101 117 125 125 111]
 [123 125 125 125 125 125 117 111 123 124 122 125 124 125]
 [ 83 115 124 124 121 109  99  98 110 123 124 124 124 124]
 [ 66  73 122 124 104  98  99  97 111 124 123 123 124 124]
 [ 64  65 117 115  97  98  98  95 119 122 123 123 124 124]
 [ 65  64  96  97  96 100  94  97 121 121 122 121 122 123]
 [ 64  65  87  91  96  98  92 112 121 121 122 121 122 122]
 [ 65  69  91  91  94  95  96 111 115 114 114 115 114 115]
 [ 69  79  88  90  89  89  92  99 100  99 100  99 100 101] ]
0.0672 (88, 165) (116, 179)
94
[ [111 108  58  24  21  21  21  21  22  20  21  22  22  22]
 [ 17  19  20  21  21  20  21  21  21  21  21  22  22  22]
 [ 19  20  20  21  21  22  22  21  22  21  22  22  23  23]
 [ 20  20  20  21  20  22  21  21  21  22  21  22  23  23]
 [ 20  20  20  21  21  22  22  22  21  22  22  22  22  22]
 [ 21  21  21  20  21  21  21  22  21  22  22  22  23  23]
 [ 20  20  21  20  21  21  21  22  21  22  21  22  22  22] ]
0.0624 (170, 133) (196, 146)
9
[ [62 61 58 56 57 55 55 56 56 55 54 56 56]
 [56 56 56 56 56 56 55 56 55 55 54 54 55]
 [55 56 55 55 55 55 56 55 55 54 54 55 56]
 [55 54 56 53 55 53 55 53 55 55 55 54 55]
 [54 56 54 53 54 54 53 55 54 54 54 53 55]
 [54 54 55 54 53 54 53 53 54 54 56 55 54] ]
0.0576 (75, 149) (99, 159)
67
[ [ 41  40  40  39  41  39  40  38  39  39  49  59]
 [ 40  40  40  40  39  39  40  40  40  39  41  39]
 [ 40  40  40  40  39  39  39  38  39  39  39  39]
 [ 41  40  40  39  39  39  39  39  39  39  39  39]
 [105  88  71  58  53  45  44  42  42  40  49  53] ]
[]

Is it clear which one is not an apple? It is not simple because I believe that one apple, touching the pot, was quite occluded by leaves. Also the smooth surface could be a chair behind in the very left rectangle.

The current plan is to switch back to higher resolution (millimeters) and try to fit a sphere there.

7th December 2014 — Probabilistic Sphere

I did not expect that for version 0 (the simplest thing which can possibly work) I will need strong weapons like probabilistic matching. Because my old lecture is in Czech I will briefly review it in English for sphere:

Suppose you have a set of 3D points where at least 50% belong to the apple surface. How to recognize, which points belong to the apple and which do not? If you randomly pick one, there is 50% chance that you succeed. If you pick two points the probability multiply so 25%. It is necessary to have four points to define a sphere in the space (the points are on the sphere surface). So the probability is 1/16th=6.25%.

Now comes the trick. We are living in the world of powerful computing machines and it is no problem to repeat our try 100 times or more. What is the chance that after 100 repetition we find 4 surface points? It is 1-(1-0.0625^100), which is 99.8% and that could be enough for the beginning .

So is it really that easy? Well, sure enough there are always some drawbacks. The first one is recognition that your sphere fits to the 50% of apple surface. What you can do is count distances of each 3D point to surface and if it is within limit you pick the sphere with the highest number of points in this zone.

Do you want so know my first result? It was:

center = (0.018495501893939396,-2108163884176161.0, -146098213063779.62) 
radius = 2.11322020869e+15

… so radius with 15 zeros so something like a quoter of light year!? The reason is that we are computing with limited precision and due to rounding 3D point within few millimeters will fall into the same double precision number. So even there were 99.7% of 3D point on the surface within the limit it was surely not what we were looking for.

I added new parameters minRadius and maxRadius set to 3cm and 15cm respectively to fix that. Here is the code diff.

There is another drawback with probabilistic algorithms — they sometimes fail and it you keep them purely random you may get different results every time you call them. For that reason every apple detector has its own random generator with random seed set to zero:

self.random = random.Random(0).choice

Now how you can get a patch with 50% „apple surface points”? You can use already mentioned MSER, or what I learned on „Computer Vision after 20 years” lecture — just brute force in these days. Write some quick algorithms/filters which will quickly reject non-apple image window. And then you can try even all possible windows. Even average PC has typically 1024 graphic processors which can do the filtering job. The windows which remains you can then throughly analyse with methods like the one mentioned today. More over if you are in production you can collect tons of reference data and your program can learn from them over time … give up humans .

Note, that an apple is not sphere. It won't fit perfectly. If you want to, you can find some parametrized 3D model, so instead of 4 points you will need probably few more, to define it better. But on the other hand we should not focus on laser only. If you add camera we have pre-selected set of 3D points and we can classify their color too. So red points are OK, and brown/green are not (if you have a leave, unluckily twisted in sphere shape, you will not distinguish it from an apple via laser).

„Final” note about my 3 apples. I added extra function saveAsCloud so you can analyse just a patch in Cloud Comapare program (see diff). The nearest "apple" is probably just leave touching the scaner — in the last experiment we put it very close. So with version of code available at https://github.com/robotika/applebot you will see this result:

fit sphere with more than 80% of points (2cm tolerance)

12th December 2014 — URCaps

A few days ago I received an advertisement from Czech PR company working for Universal Robots — successful Danish robotic company. I already know about their product UR5 (robotic arm capable to carry 5kg). You also need an end effector (gripper), and that is something what was missing in UR portfolio. They decided to do an interesting step: they created URCaps website, where you can find various U R Capable add-ons.

URCaps

Why I am mentioning this on the AppleBot blog? Well, even from the very begging I was thinking of UR5 as potential platform for picking the apples, but I did not get confirmation yet, that the arm will survive transport between stops for picking.

15th December 2014 — PoE IP Camera

Last Tuesday we received new camera for robotic experiments. It is from the same manufacture as the camera on robot Eduro. It is compact (half length) and it has two sensors — see AV3236 specification. The camera is sold for Day/Night surveillance and it has removable IR cut filter.

When we unpacked it we wondered: where is the power connector? There is none! There are 4 pins for input/output, but none for separate power. The reason is hidden in label PoE (Power over Ethernet). i.e. the camera is powered by unused lines of Ethernet cable.

You need something like power injector. AirLive passive PoE Kit looked good, but when I attached the power (18V, 24W) it did not work :-(. I supposed that the camera is for wide range 12V-30V(?) like the previous one, but it is not. PoE and Auxiliary Power: 12-48V DC/24V AC (Auxillary Power Not Available for Dual Sensor Model). You need PoE standard voltage which means 48V. The camera is still working on 32V (that is the very edge), but 48V is the recommended voltage.

I have seen first H264 video and some pictures from this sensor, but I do not have the power adapter now to run it and show it you too — so at least some photos:

Camera in box

Dual Sensor

Back view

16th December 2014 — Dual Sensor

Now only a quick info about camera status. I bought 48V power adapter and when I connected power to injector and Ethernet cable to camera it did not work. Again. Or to be precise there is not way of telling if it is working or not. The LED for Ethernet will start to blink only when Ethernet cable is also connected on the other side (to router or laptop). So make sure you have connected everything and then it works. Fixed.

The second issue was IP address of the camera. If you have Windows OS then probably the simplest way is to install AV200 Software and click and click. The good news is that old command/URL to get JPEG image is still working:

http://192.168.1.6/img.jpg

Which sensor is used depends on your settings. Default is Automatic (for bright light it is using Day sensor and for darkness Night sensor), then Day, Night and Dual Channel Streaming. Here are first pictures for day and night manual setting:

Day sensor

Night sensor

So it looks like the left eye is for night mode. The next step would be to test streaming of both sensors — I am curious if there is still a chance to extract individual images from both sensors. We will see (later today at school).

p.s note, that by default day sensor has resolution 1024x768 and night sensor 640x480

17th December 2014 — 2nd 3D-Scanning Experiment

Yesterday we repeated the 3D scanning experiment. This time in combination with new camera so we have also reference pictures with timestamps. Again, there were several new things we learned.

First of all it is the new Dual Sensor Camera AV3236DN. I did not succeed to automatically record H.264 video streams from both cameras, so fall back was a single image every 10 scans (see diff).

It was quite hard to find Camera API. There are several manuals, papers, presentations pointing to non-existing page :-(. At the end I found it here, but I doubt it is persistent link. Look for ArecontVisionAPIandCameraWebPageManual6.23.2014.pdf. Here are some interesting hints:

http://192.168.1.6/livevideo
http://192.168.1.6/h264stream
http://192.168.1.6/h264f

Once I typed h264stream in my browser it generated like 100 windows to save data — not good. The h264f is shortcut for H264 frame, but I was receiving I-frames only and no P-frame even when I was changing iframe=0 parameter (OT you can be interested in my experiments with H264, but it is in Czech only at the moment). So it needs more calmer time for investigation and that was not yesterday.

Another issue we had was related to remission data from laser scanner TiM55. I remember Jirka was trying to get it work under Linux (see his website). The easiest way was to use supported software to enable remission (checkbox RSSI), and then switch Ethernet cable back to my notebook.

There were extra 3 elements in output array (instead of expected 580 like on older USB laser scanner there were 583), and now I see on Jirka's page that "B not defined". Yes:

A9', 'B2', 'B2', 'AC', 'AC', 'AC', 'B2', 'AF', 'AC
', 'A9', 'AC', '0', '1', 'B', 'not', 'defined', '0', '0', '0']

… that's the same problem.

OK, I have to get ready for early morning meeting, so at least some results. Robotics should be also fun, so thanks Standa for volunteering

Distance data

Remission data

Day camera

Note, there is minor change of log2pgm.py in order to handle both range and remission data.

p.s. based on the timestamps of taken images every 10 laser scans it looks like we really running at 10Hz, but the sensor should support 15Hz, i.e. better resolution in X coordinate … next time.

18th December 2014 — applethreshold.py

There is new interactive tool on github now: applethreshold.py. It is modification of threshold.py where I am interested only in narrow space slice. The apple has some expected size, say 10cm, and in the depth image we can see only front face. This defines the slice width to 5cm. All other distances are drawn with white color.

The laser scanner is measuring in millimeters and we use division by 5 for gray conversion, so the 5cm corresponds to integer appleSize=10. Here are two space slices with apples at 91/2 cm and 98/2 cm distance:

Slice for 43-48cm

Slice for 46.5-51.5cm

What is it good for? Well, I was not very happy with brute force experiment, so I am looking for an alternative solution. This is just a visualisation tool — the algorithm should look for uniform blobs of given size.

19th December 2014 — Crazy Ideas

I must be influenced by the book Naked Sun I am just reading. There are plenty of Solarian robots working in apple orchards … I was looking for Christmas present two days ago — a telescopic tool for picking apples, and I end up on this Czech website about work and travel on New Zealand. And then I realized, that I do not have to wait till September for real apple picking test. There is new world where the picking season starts at the end of January and continues till April .

Another piece of jigsaw puzzle is EXACTEC, Czech distributor of Universal Robots. There is a chance that we will have their UR5 robot for testing for a week. The goal is version0 i.e. to autonomously find an apple and pick it from the tree (because of the winter I would be probably rather Christmas tree). And yesterday I learned that on 20th of January 2015 I should have a presentation about my future research … so why not to join these two, and present the first results on the real machine?! Yes, mission impossible, but we are used to it, right? Did we have more time for any other „competition”? There is one month remaining (well, half of that are holidays), and it is strong motivation to win.

So instead of H264 codec I was looking rather on UR5 specs and communication protocol. Based on the English manual there should be Ethernet, which is now probably the simplest way of integration (both laser scanner and IP camera are connected to the laptop over Ethernet). Moreover I already found some Python examples how to control the robotic arm (see Zacobria Robot community forum), and it looks like fun. Is it crazy enough?

21st December 2014 — Memories and UR5 API

Yesterday I had no computer day so besides packing of a few Christmas presents I re-read my very old book about robots (I bought it when I was on elementary school). The title is Roboty slouží člověku (Robots Serves Humans), Surý, Rempsa 1982. I remember that I was a bit disappointed that time (30+ years ago) and probably did not understand it very much.

If nothing else it is clear now, why UR5 has 6-DOF (six degrees of freedom). If you want to be able to hold a tool in given position (x,y,z) you need 3-DOF. But if you also need 3D orientation (roll, pitch, yaw) then you need obviously six degrees of freedom .

My second memory, related to robotic manipulators, is connected with Probabilistic Path Planner (PPP). In particular I remember talking to prof. Overmars in 2000 (Eilat, Workshop on Computional Geometry), when he was explaining unnecessary movement issues with probabilistic planners with large number of DOFs (snake-like robots with 20 or even 50 DOFs).

The third memory is relatively fresh. Last year I was playing with hexapod FireAnt having 25 servos, i.e. 25-DOFs. Each leg had 3 servos and it was necessary to compute leg transition from one position to another … and the same holds now for UR5 too. So the distinction between commands moveJ and moveL is clear now .

UR5 API

Position of 6-DOF robot is defined by 6 numbers. In the case of UR5 these numbers correspond to the angles of each joint. UR is using term waypoint for this 6-tuple.

Next, if you have a motor, the simplest control curve is to define acceleration and maximal speed. The speed profile then looks like trapezoid.

OK, so we have waypoints and standard control for motor speed. Now it starts to be a little bit more tricky. What if you combine two or more waypoints? Say when you want to move from one position to another? See page 117 of UR5_User_Manual_GB.pdf:

moveJ — will make movements that are calculated in the joint space of the robot arm. Each joint is controlled to reach the desired end location at the same time. This movement type results in a curved path for the tool. The shared parameters that apply to this movement type are the maximum joint speed and joint acceleration to use for the movement calculations, specified in deg/s and deg/s2, respectively. If it is desired to have the robot arm move fast between waypoints, disregarding the path of the tool between those waypoints, this movement type is the favorable choice.
moveL — will make the tool move linearly between waypoints. This means that each joint performs a more complicated motion to keep the tool on a straight line path. The shared parameters that can be set for this movement type are the desired tool speed and tool acceleration specified in mm/s and mm/s 2 , respectively, and also a feature. The selected feature will determine in which feature space the tool positions of the waypoints are represented in. Of specific interest concerning feature spaces are variable features and variable waypoints. Variable features can be used when the tool position of a waypoint need to be determined by the actual value of the variable feature when the robot program runs.

Note, that there exists also moveP, which will move the tool linearly with constant speed with circular blends, and is intended for some process operations, like gluing or dispensing.

I suppose, that combination of moveJ and moveL should be enough for control of applebot-ver0 …

22nd December 2014 — Apple Generators

On Sunday evening I did one more experiment. An apple has shape close to sphere so the central part has to be closer than the points on the edge. That is what you should see after this diff.

two levels threshold

Here you can see that the center part the apple is gray while the area around it black. Note, that this is also true for the pot below.

Then I went back and tried to detect dense areas, i.e. if you have window 10x10 pixels that most pixels are occupied. Note, that all you need to do is convolution with matrix filled with ones (you have to change image that apple pixels are 1 while the rest is 0). With applethrehold.py you can now see both windows (threshold and dense areas).

The last step was change of dense areas into generator of interesting positions and integration with the apple sphere matching — see the diff.

Does it sound/look good? Well, the result is a bit disappointing

dense areas apple detections

You see that also the pot and the chair has 3D surface similar to sphere. There will be probably no such smooth large areas in the outdoor tests, but still something which should be solved soon.

This is crazy:

winSizeY 4 (271, 380, 3)
1.00: (0.034, 0.847, -0.035) 0.085

so this part of chair fits 100%?! There is narrow window because of the 0.899m distance, but … I can add there also the upper limit — not all point should fit perfectly. It is projection of sphere with bounding rectangle. Expected fit is given by area, i.e. for radius=1 is the area PI and square is 2x2 = math.pi/4. = 0.7853981633974483.

Here is result with upper limit:

upper limit for fitting sphere points

and here is the code change. There is still problem on the border (and always will be), and also the tolerance to apple view blocking leaves is lower.

4th January 2015 — Recording H264 video

Well, I was a bit sick last two weeks … I was not obviously following recommendation: An apple a day keeps the doctor away . It is still not very good but I miss playing with Applebot .

I would start with experiment with the new camera today. You can find now on github simple script dualcam.py, which is recording H264 video stream. I was actually very close to the working solution. The missing parameter was ssn — stream identifier. If you do not specify it you will always get I-frames only.

http://192.168.1.6/h264f?res=half&qp=20&ssn=13
http://192.168.1.6/h264f?res=half&qp=20&ssn=13&iframe=0

[SOLVED]

The second task is to enable day and night sensors. It is necessary to enable given input:

http://192.168.1.6/get?daynight
http://192.168.1.6/set?daynight=dual

and then you can download separately color or monochrome images:

http://192.168.1.6/image?channel=mono
http://192.168.1.6/image?channel=color

Here is the first attempt to set the mode and record H264 video … but it does not work, so far. If I use day or night mode then video stream is OK. But if I select dual then channel=color and channel=mono does not seems to have any influence. Maybe mix with ssn?

9th January 2015 — stereo_match.py

Almost two weeks ago I was playing with following idea: If I have several images collected from linear motion, I should be able to get some stereo vision results?! … and sure enough there is directly example in OpenCV2 (see hint from stackoverflow).

It is necessary to slightly modify the stereo_match.py source code, that it will read your images instead of prepared demo (BTW the source was in my case in c:\opencv\sources\samples\python2\stereo_match.py) and as result you will get scene of polygonal patches in PLY format.

Note, that you can open it with already mentioned CloudCompare and you will get something like this:

Stereo Apple

… unfortunately the static image is quite boring. If you view it in 3D you can well recognize the shape of pot, two apples and also small citrus in front .

11th January 2015 — matplotlib

The topic today was supposed to be debugging laser apple, as I was trying to find out what is going on in the apple 3D recognition from the laser scans. There were bits I expected, like blended borders when readings on the edge of apple are non-sense. They correspond to some averaging of distance to the apple and distance to the background. Other details were not expected as for example variation of laser readings.

At first I wrote a simple cutting utility, which is based on OpenCV2 sample mouse_and_match.py. You can click on the image, select rectangle and then do something with cut patch. Current version opens extra window with 16x times zoomed image and dumps gray values for displayed image. The latest version dumps also data of the original laser scan and its corresponding remission. For more details see cutter.py history.

So now I can see the numbers. There are only a few of them — even for an apple 30cm from the scanner I got like 8 readings, which corresponds to approximately 4cm in height … strange, I expected more apple-related readings.

The second surprise was how bad the reading were. There were jumps like 2cm on the surface of the apple! I tried to visualise it and that's the moment when matplot comes into the game. After installation on Win7 I had also install six, dateutil and pyparsing, but that was surely worth it .

Here is the first plot for the laser scans of the apple:

detail laser readings for an apple

The Y-axis is distance in millimeters and you can see for example on the red plot 1cm oscillations. This is maybe a better visible on the graph with artificial offset 1cm for each plot:

scans with 10mm artificial offset

And one more graph, when for both axis correspond to distances in millimeters:

these plots should be semi circles

The artificial offset was set to 5mm here in order to fit all plots in the image. Do you see the plots as approximation of various semi-circles? Well, maybe …

16th January 2015 — The First Tests with Universal Robot UR5

Yes! Believe it or not we have now the robotic arm UR5 from Universal Robots. Just for a week (for the Tuesday Jan 20th show), but still it is a great!

I am currently exhausted and still sick and I should go home, but I am waiting for completion of devices download for SOPAS, which I need only to change IP address of our laser scanner :-(. We already did it some time ago, but it was probably not saved so now it works on 169.254.225.156, which is not very useful when camera, robot UR5 and my laptop run all on 192.168.1.x network … just to be patient, which is hard for me now.

So where are we? We have the robotic arm working = it scans, goes to desired position, close the gripper, returns home. Smoothly, nicely . If you would like to see the code, it is on github under ur5.py.

Camera is recording, so there should not be any problem.

Laser is an issue now. First the IP but also the mounting has to be revised. There are very bulky cables and the first version of the gripper with sensors did not take into account wide radius of the arm.

Time to go home …

gripper - front

gripper - side

UR5, work in progress

… at home …

On the way home I was thinking about several steps we already completed, but I did not mention them. I am more relaxed now so it should be easier to do the revision:

Inverse kinematics

Well, it was not necessary to do ANY computations . There is magic p prefix to coordinates send to UR5 so if you send for example:

movej( p[0.139, -0.065, 0.869, -1.311, 1.026, -0.869] )

then the parameters are not angles in radians but they are x, y, z, rx, ry, rz instead. And because we need angles fixed for version 0 there is no need to worry what exactly rx, ry, rz means. You just turn the wrist to the orientation you need and change only the first three parameters x, y, z.

Server vs. Client

When we visited Exactec I learned that the UR5 is so universal that it also supports many types of communications including one, where the robot is only a slave/client. You can open sockets to other servers, run several threads etc. I am glad that also the other way works, i.e. I use UR5 as server and my notebook is client sending requests. For this kind of communication is dedicated port 30002. Note, that you can change IP address of the robot, but it is highly recommend to reboot the robot after this action . After reboot it worked.

Status Info

I was wondering how to get regular update about current robot joints position and it turned out that all you need to do is listen . There is a binary protocol containing 1247 bytes per packet, where you will find everything from positions, targets, speeds, Cartesian coordinates, voltages, currents, temperatures …

At the moment I am using only absolute speed of all joints as terminal condition (reached desired position).

Logging

I fixed my mistake that in Liberec I did not have logging ready. Because we forgot there power cable (do not laugh ) I was using only bits remaining in my console. Now I have almost 100 files from today testing, so crack what is 64bit timestamps should be trivial. And it was (see diff). So it is probably time in milliseconds since reboot. The differences are like 12-13ms, i.e. update is approximately 80Hz (???). I will have to measure absolute time as reference tomorrow.

17th January 2015 — UR5 Video

Here is the first video with UR5 with camera and laser scanner in the „hand”:

The corresponding demo code commit is here.

You may notice couple details: first of all the laser scanner is rotated by 180 degrees. There is the yellow Merkur part (something like Mechano), which was quick and dirty solution, but it works:

I quite like it for rapid prototyping .

Second, the hand does not look like on pictures with Merkur above. I was getting several times protective stop, position deviates from path Wrist1 until I realized that the hand is actually self-colliding!

Self colliding robot

The solution was to change all three joints to this configuration:

Upside down configuration

What was interesting that X,Y,Z as well as rotation angles were the same as before! Well, I was warned but you do not quite believe it until you see it . So letting robot solve inverse kinematics is good, but there could be more than one solution and only the joints space defines the robot pose unambiguously!

What else? You can hear the clicking sounds near the apple, as the robot is trying to pick it and then to drop it. I repeated the test six times and in four attempts the gripper did not work?! I am not sure if the command was lost or what was the problem? So one task is to review UR5 output stream if the I/O is confirmed. It is nice to have all the log files (sent commands included) where you can see that in all tests the gripper was supposed to close and open.

Well, I do not see anything in Tool Data, except small current change at one moment (diff). There are also some Masterboard Data (diff), and I/O bits changed from ['0x20000', '0x0'] to ['0x10000', '0x10000'] … hmm, interesting. Yes! I checked the second log file and the bits did not change. Just double check that it worked in 6th test … yes . OK, so it is probably necessary to verify these bits and repeat the command until they change.

There was an interesting moment when my Ethernet cable was accidentally disconnected. UR5 stopped to talk to me. It looked like that it is necessary to send some command to UR5 first and the it starts to send the data again.

I did not mention that the video is fake!!! — if you followed the github link you would see it, but who would do that, right? The data collection was running so you can see me recording the video:

I see you you see me ...

The laser was running all the time so top-down scan corresponds only to part of this distance map image:

Distance image from laser scanner

Do you see there the apple with cable? In the last experiment I moved away my notebook and I slowed the scanning 4 times (original speed was 0.1m/s). One pixel corresponded to 1mm (laser has update 10Hz) and after change to 1/4th … do you see the difference?

4times slower scan

I would guess that it is only two times slower, but the log file should clear this up.

p.s. timestamp 14000-8547=5453 corresponds in reality to 43 seconds (images pic_150117_132421_000.jpg to pic_150117_132504_039.jpg), so 1 step = 8ms??? Strange. Based on the size of log file the update looks like 20Hz. Maybe some other documentation will clear this up.

p.s.2 now I should uncomment the apples = findApples( APPLE_SIZE, scan ) and see what would real detector return + do transformation to 3D with some offsets …

20th January 2015 — Applebot UR5 ver0

It is working now — see corresponding diff. It is not perfect as recorded fake over weekend, but it does the work autonomously:

scan the scene
search for the apple in recorded 3D data
compute desired 3D pose
navigate hand towards the apple
close gripper
move towards drop zone
open gripper

I started to record from both cameras and as you can see I had a witness last night:

Milan watching the demo

Note, that it is not perfect yet. In particular it is necessary to calibrate rotation around Z axis. I used only 3D offset and that is not sufficient. Slight rotation of the scanner + imperfect mounting is probably source of couple centimeters error, which depends on the position on the table (BTW thanks Milan for nice „apple holder”).

The algorithm is quite simple now. It is using dual threshold defined by apple size and scans with 5mm step from 20cm to 50cm. It looks only for objects (contours) of expected size (area parametrized by motion step and distance) in given 3D slice. Only the first occurrence is accepted now (for higher reliability for the today's demo).

If you see it working you may find it trivial, but it was a bit „jumpy road” towards this intermediate goal.

Over weekend I was reading about similar apple harvesting projects, and I read the Chinese article again. Why did they use 9kg SICK LMS211, with so bad accuracy (10mm +/- 15mm)?! Well, it was not so bad — the TIM551 has Systematic error +/- 60mm and Statistical error +/- 20mm! So the fact that I had problem recognize smooth 3D sphere shape in 3D cloud from small patch is expected. :-(

I also checked specs of our first scanner LMS100 (from SICK Robot Day 2010) and it promise Systematic error ± 30 mm and Statistical error ± 12 mm. Also it can scan at 50Hz scanning frequency with 0.25 degree resolution. The bad news is weight 1.1kg without cables and operating range starting from 0.5 meter. So we will probably stay with the smaller TIM551. On the other hand maybe it is just a typo, as TIM551 has operating range from 0.05 meter and we used LMS100 mostly in this close range — see data sheet.

This remains me the issue with scanning frequency of TIM551. It should be 15Hz, but I was getting 10Hz only. Sure enough it is mine fault, due to historical reasons. It was probably not possible to switch LMS100 to scanning frequency 10Hz, so we had to add there sleep (see Eduro laser.py code). So by removing this sleep we may get now 50% faster scanning rate .

Another eyeopener was with UR5. Do you remember my comment that I would expect the apple from the 6th test on Saturday much bigger (4 times) then in previous experiments? Well there is a global slider named Target Speed Fraction, which is probably used mainly in manufacturing, where you first test your assembly line in slow motion and then you increase to 100%. I was operating the arm with touch screen control, so I set it to slower speed (41%). But this control influences also external commands! So in reality I was not scanning with speed 0.1m/s but rather 0.041m/s. For the last (6th) test I first rebooted all machines, just to be sure, so this slider was back on 100% and slow speed 0.025m/s was in reality just approximately 2 times slower then in the first test .

Shadow Robot Hand

24th January 2015 — The Hand

I went across this article last weekend when I was preparing my presentation. It is about space research but see the picture! And believe it or not it is not science fiction . You can contact Shadow Robot Company and get quote for their robotic hand. I was interested in combination with UR5 and they offer even this combination (also with UR10). Options are left/right hand but also Lite Hand, which would be the option to combine with UR5. The "normal" hand weights 4.3kg, which is very close to UR5 payload (5kg) while the "lite" version is 2.4kg only.

As you would expect it is not cheap toy. But if you take into account that you can control all part of all fingers and you have force feedback … one day I would like to test it .

25th January 2015 — Standa and UR5

This is a video recorded shortly before we packed UR5 back into the box. Standa wanted to try to program it from control panel and you can see that he was more successful than me with autonomous navigation . Note, that all sensors were running, in particular you can see TIM551 scanning which you would not see with bare eye.

30th January 2015 — The Others (part1)

According to a nice survey article Harvesting robots for high-value crops: state-of-the-art review and challenges ahead, Bac, Henten, Hemming, Edan, Journal of Field Robotics (2014) … performance of harvesting robots did not improve in the last three decades and none of these 50 robots were commercialized. (A project was included in the review if and only if a complete functional system was built and reported in an English written conference paper or peer-reviewed journal article.)

Well, that is a bit discouraging. There were only five (!) project for apple harvesting and from them only two robots were autonomous. It is possible that situation changed last year or that some project was not published or authors overlooked it, but still … I expected more projects.

The first presented autonomous apple harvesting robot (AAHR) is from Belgium and you can read about it in popular article for example here and here is scientific paper. I was told that the robot worked well, but the remaining problems were increasing productivity, ensuring safety (according to standard regulation in open field), and the price bringing it to the market. There was in the end no company willing to take the risk. At the moment there is new group continuing on the work in this research area.

The second presented AAHR is from China Design and control of an apple harvesting robot, but I did not track that path yet.

There is new promising 2014 project on New Zealand — see this article. A new startup company with already successful kiwi harvesting robot was supported by government to commercialize the technology, and the goal is also to pick apples. The project just started and it is for 4 years.

Finally note for myself , that all we tried to present with our experiment here I found yesterday in 15 years old article: A vision system based on a laser range-finder applied to robotic fruit harvesting. The laser technology was not so widespread that time but they did very nice work in recognition of partially occluded spherical fruits. The inputs were distance and reflectivity measured by laser scanner.

contact form.

Jessica

2014-11-03T00:00:00Z

Here are the first pictures:

box

minidrone

battery

The drone was lend from Czech Parrot distributor: http://www.icornerhightech.cz/

Parrot Minidrone

… and here is the link where hopefully will be SDK from Parrot one day:

https://devzone.parrot.com/projects/show/oss-rolling-spider

Currently you will find there: Note: this is not the Software Development Kit (SDK) for developing applications to control the drone from a remote device. The SDK will be released at a later time.

Unfortunately we have tight dead-line, because Tour the Stairs is by the end of November 2014 and the situation with SDK probably won't change.

Bad news

Rolling Spider uses Bluetooth 4 = BLE, which means Bluetooth Low Energy. This sounds like an interesting new technology but for example my Windows 7 does not support it (update: it looks like there are some new drivers). On the other hand the other notebook with Windows 8 behaves like it could potentially talk to the drone.

Rolling Spider comes with application Free Flight 3, which is nice, but … in order to talk to the drone you need not only BT4 but also your device has to be in a relatively short list of supported devices:

http://blog.parrot.com/2014/07/22/devices-compatibility-with-minidrones/

Good news — btsnoop_hci.log

There is a hope and it has name btsnoop_hci.log . What is it? Since Android 4.4 there is a new tool for Bluetooth capture communication (see this article). Just check in Developer options option Enable Bluetooth HCI snoop log and whole communication is logged into the file btsnoop_hci.log. Cool isn't it?!

No idea what to do with it? Well, there has to be coded info why drone refuses other phones, commands to fly, termination of communication, etc. At the moment I have couple logs from two different phones. If you want an example here is one log from Samsung S4 (well I am not sure what kind of "secret" information is hidden there). It was just on/off and once I accidentally touched the display and the propellers turned a little … and it has 140kB, sigh.

The parsing is relatively simple. It contains timestamps, how many bytes are transfered and in which direction. Here is new Jessica's repository on github with simple parse.py.

If you have phone with Android 4.4 and would like to repeat what I did, here are two important details:

the default phone menu does not contain Developer options
btsnoop_hci.log is not necessarily at /sdcard/btsnoop_hci.log

The first thing was simple — the answer is for example here. It is necessary to press seven times in About phone otherwise gray Build number and you will see You are a developer!

The hint about exact absolute path was written in already mentioned article. In reality it was /sdcard/Android/data/btsnoop_hci.log, but it can be different on every system. In particular it is necessary to look in file /etc/bluetooth/bt_stack.conf and there you will find it. If you have problem how to look at this directory you can use for example Total Commander.

AR Drone forum: SDK support for Bebop and Rolling Spider?

Blog

5th November 2014 — BLE, BLE, BLE

It is too early and there are no suitable „ready to use” tools. I am giving up combination "Bluetooth 4 BLE Python Windows 7". It looks like there is only one Python wrapper supporting Bluetooth Low Energy for Linux (bluepy), but that is basically wrapper for bluez. It could be a source of inspiration though like list of services UUIDs.

The way to go now is to get running BluetoothLeGatt example for Android 4.3+ in Java. GATT seems to be an important word — Generic Attribute Profile used for BLE. It would be just too simple to use the sample: Note: At this time, the downloadable projects are designed for use with Gradle and Android Studio. Project downloads for Eclipse will be available soon! (taken from samples page).

And what I am trying to do? Well, it would be a bit useless to fully understand the btsnoop_hci.log if I would not be able to replay it. So that is the plan — record some minidrone action with logging facilities and then replay it on the same phone, where it should be hopefully work. And then, in the next phase, try to understand the command details.

p.s. obviously I am not the only one, who is looking for Python BLE libraries…

7th November 2014 — Android Sample

I have to write down some notes, because you would not be able to tell that there was some progress (especially if there are no new commits on github). The good news is that I managed to get working Android Bluetooth LE Sample. I was not „click and go” — instead that was a kind of blind fight with installation, drivers, etc.

At the end I decided to try Android Studio. I am still not sure if I was supposed to use studio.exe directly or studio.bat, but now I have set path to Java SDK (the bat wrote: ERROR: cannot start Android Studio. No JDK found. Please validate either ANDROID_STUDIO_JDK, JDK_HOME or JAVA_HOME points to valid JDK installation.) and I also overcome problem in studio itself:

Error:Unable to start the daemon process.
This problem might be caused by incorrect configuration of the daemon.
For example, an unrecognized jvm option is used.
Please refer to the user guide chapter on the daemon at 
   http://gradle.org/docs/1.12/userguide/gradle_daemon.html
Please read below process output to find out more:
- - -  - - - - - - - -  - - - - - -  - - - - - -
Error occurred during initialization of VM
Could not reserve enough space for object heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

The solution for this was found on stack overflow: to add something like this to your gradle.properties file in the project:

org.gradle.jvmargs=-Xmx512m -XX:MaxPermSize=512m

I did not have any file named gradle.properties, but details where such a file could be you can find in gradle documentation.

So what the Bluetooth sample does? It scans for BLE devices, if you select one it connects to it and list available services, and if you select service it can read given server (e.g. minidrone) characteristics. I probably mismatched exact Bluetooth terms, but what I mean is that you can connect to the drone and ask what it can do. And surprisingly enough 80% (?) of initial communication is identical to Free Flight 3!

The conclusion is that it is not only „decoding Parrot protocol” but rather understanding GATT and Bluetooth LE protocol (which is good and bad news in one). For me this is completely new area, but the motivation is relatively strong to learn more. An interesting online book with reasonable explanation is here, Chapter 4. GATT (Services and Characteristics).

A funny note at the end of this update … as I am decoding communication bottom up it is fun to discover messages like this (two different phones): [2, 64, 32, 17, 0, 13, 0, 4, 0, 9, 11, 22, 0, 71, 97, 108, 97, 120, 121, 32, 83, 52] and [2, 64, 32, 27, 0, 23, 0, 4, 0, 9, 21, 22, 0, 76, 106, 32, 67, 114, 101, 97, 116, 111, 114, 32, 40, 71, 97, 108, 97, 120, 121, 32]. Do you see there some similarity? And what about [2, 64, 32, LEN=27, 0, LEN=23, 0, 4, 0, 9, LEN=21, 22, 0, 76, 106, 32, 67, 114, 101, 97, 116, 111, 114, 32, 40, 71, 97, 108, 97, 120, 121, 32]. So it is envelop of envelop for envelop for string. Almost half of transmitted bytes are length of sent buffer.

9th November 2014 — GATT, Services, Characteristics, …

It is quite hard for me to read through Bluetooth LE specification. The bits slowly fit into the whole image, after many reading attempts. Everything is written there just my brain somehow refuses to understand it .

Now I would like to say that the best start is probably wikipedia Bluetooth Low Energy article. Note, that you have to understand the BLE concept if you want to understand Parrot minidrone protocol, so that's why I am trying „so hard” .

GATT, Services, Characteristics, … these are all keywords used in BLE. GATT is an acronym for the Generic Attribute Profile, and it defines the way that two Bluetooth Low Energy devices transfer data back and forth using concepts called Services and Characteristics. It makes use of a generic data protocol called the Attribute Protocol (ATT), which is used to store Services, Characteristics and related data in a simple lookup table using 16-bit IDs for each entry in the table. (source) Clear?

I cannot find now mine „eye opener” image/table so at least one more complicated: source.

GATT based on L2CAP

So GATT is based on existing L2CAP (Logical Link Control and Adaptation Protocol) which is something similar like UDP on Internet communication (and it is not available under Microsoft Bluetooth Stack, as far as I understand it). For night reading I would recommend Bluetooth for Programmers PDF from MIT — a bit older with many TODOs, but still good enough. There you will learn about L2CAP.

Does it help? Well not really. It means that if I would be able to send packets via L2CAP I should be able to replay logged data as-it-is. Maybe. But if I want to use GATT available in Android, for example, then it is necessary to understand the protocol in more details and go to the ground what each byte means.

I would pick some (for me important) sentences from wikipedia:

Bluetooth Smart is not backward-compatible with the previous Bluetooth protocol. Bluetooth Smart uses the same 2.4 GHz radio frequencies, but it uses a simpler modulation system. (i.e. not good idea to convince old hardware to talk with BLE)
Write operations always identify the characteristic by handle, but have a choice of whether or not a response from the server is required. (So it is not enough to know characteristics by its UUID, you need to find the handle.)

Now back to the minodrone. If you run Android Bluetooth LE sample, you will find a big list of available characteristics (32+32+7=71 in total). The complete list is in modified SampleGattAttributes.java. But how to find which UUID corresponds to the „magical command” for motion control, mentioned sooner? After some experiments with write it looks like that the assert with array [0x12, 0x0, 0x4, 0x0, 0x52, 0x40, 0x0, 0x2] contains the 16bit handle [0x40, 0x0].

How to write characteristics? This part is missing in the example, but you can find the answer here.

This was my hack (modification of DeviceControlActivity.java):

//hack mBluetoothLeService.readCharacteristic(characteristic);
// hacking
byte[] value = new byte[5];
value[0] = (byte) (0x11);
value[1] = (byte) (0x22);
value[2] = (byte) (0x33);
value[3] = (byte) (0x44);
value[4] = (byte) (0x55);
characteristic.setValue( value );
mBluetoothLeService.writeCharacteristic(characteristic);
// end of hacking

and changes of BluetoothLeService.java

public void writeCharacteristic(BluetoothGattCharacteristic characteristic) {
    if (mBluetoothAdapter == null || mBluetoothGatt == null) {
        Log.w(TAG, "BluetoothAdapter not initialized");
        return;
    }
    mBluetoothGatt.writeCharacteristic(characteristic);
}

If you look in btsnoop_hci.log you will find there 0x11, 0x22, 0x33, 0x44, 0x55:

[0x2, 0x40, 0x20, 0xc, 0x0, 0x8, 0x0, 0x4, 0x0, 0x52, 0x22, 0x1, 0x11, 0x22, 0x33, 0x44, 0x55]
[0x2, 0x40, 0x20, 0xc, 0x0, 0x8, 0x0, 0x4, 0x0, 0x52, 0x25, 0x1, 0x11, 0x22, 0x33, 0x44, 0x55]

I was sending this array at first to characteristics D52 and D53 (the only difference is 0x22 and 0x25), but then I tried Axx characteristics (there are 32 of them) and got

[0x2, 0x40, 0x20, 0xc, 0x0, 0x8, 0x0, 0x4, 0x0, 0x52, 0x22, 0x0, 0x11, 0x22, 0x33, 0x44, 0x55]
[0x2, 0x40, 0x20, 0xc, 0x0, 0x8, 0x0, 0x4, 0x0, 0x52, 0x25, 0x0, 0x11, 0x22, 0x33, 0x44, 0x55]
…

At first moment I was quite disappointed that I am getting the same numbers as for D52 and D53 and that it is probably dynamically assigned … but I overlooked 0x1/0x0 difference . And handle is 16bit, so the treasure has UUID=9a66fa0a-0800-9191-11e4-012d1540cb8e.

p.s. do not ask me if it is already flying — it is not :-( … not yet.

p.s.2 one implementation note, which could be useful the advertisement packet is composed of a series of variable length blocks, that can appear in any order. each block starts with a length byte, followed by a type byte, followed by the data. the payload cannot exceed 31 bytes. … taken from RFduinoBLE github … but maybe it is not related.

11th November 2014 — Cracking Handles

Thanks to Šimi I realized why I was not able to find handles (16bit identification of characteristics) for Bxx services. The reason are GATT PROPERTIES. It struck me out when I saw in debugger that mProperties differ:

mUuid = {java.util.UUID830047596672}"9a66ffc1-0800-9191-11e4-012d1540cb8e"
mService = {android.bluetooth.BluetoothGattService830047490720}
mProperties = 12
mPermissions = 0
mKeySize = 16
mInstance = 0
mWriteType = 1

mUuid = {java.util.UUID830047598160}"9a66fd22-0800-9191-11e4-012d1540cb8e"
mService = {android.bluetooth.BluetoothGattService830047491960}
mProperties = 30
mPermissions = 0
mKeySize = 16
mInstance = 0
mWriteType = 1

The bit array is following:

public static final int PROPERTY_BROADCAST = 1;
public static final int PROPERTY_EXTENDED_PROPS = 128;
public static final int PROPERTY_INDICATE = 32;
public static final int PROPERTY_NOTIFY = 16;
public static final int PROPERTY_READ = 2;
public static final int PROPERTY_SIGNED_WRITE = 64;
public static final int PROPERTY_WRITE = 8;
public static final int PROPERTY_WRITE_NO_RESPONSE = 4;

So some characteristics you can write only, some can be read only, etc. And mine piece of code was in "if can read" condition:

final int charaProp = characteristic.getProperties();
if ((charaProp | BluetoothGattCharacteristic.PROPERTY_READ) > 0) { … }

(I am not Java programmer, but this bit operation would be wrong in C … I would guess that there should be & instead … this is taken from the Android Sample)

What I am up to is to find handle for all UUID, and after reading this webpage I was looking for another way: 2. The Android BLE developers chose to use UUID as the characteristic identifier in GATT API's; handles are not directly exposed to applications. Internally, Android maps UUID's to handles and uses handles to access characteristics, but apps have no need to know the handle. … so it is not a good idea to search for it in Android Java.

The good news is that you can find it in btsnoop_hci.log. I wanted to know handle for 9a66fa0a-0800-9191-11e4-012d1540cb8e (I know this one, it is 0x40). All you need is to search for "0a fa 66 9a" (reversed order), and you will find 02 40 20 1B 00 17 00 04 00 09 15 3F 00 04 _40 00_ 8E CB 40 15 2D 01 E4 11 91 91 00 08 0A FA 66 9A

… so now I have all handles of all characteristics + their properties .

12th November 2014 — Missing handles and more UUIDs

OK, now I think that I can finally generate identical output as FreeFlight3. But it was not that easy. Where was the problem? Well I could not get this output section:

02 40 20 09 00 05 00 04 00 12 C0 00 01 00
02 40 20 09 00 05 00 04 00 12 BD 00 01 00
02 40 20 09 00 05 00 04 00 12 E4 00 01 00
02 40 20 09 00 05 00 04 00 12 E7 00 01 00
02 40 20 09 00 05 00 04 00 52 16 01 01 00
02 40 20 09 00 05 00 04 00 52 26 01 01 00

The last two "packets" have 0x52 which is similar to

02 40 20 0A 00 06 00 04 00 52 7C 00 01 01 01

… and this means write 01 01 01 to handle 0x7C. So in previous section this should correspond to "write 01 00 to handle 0x116". Well, but there is no such UUID, which would correspond to handle 0x116!? :-(. Do you know the answer? I did not. The major hint was BLE_SensorTag_GATT_Server.pdf where I found Client Characteristics Configuration --- Write "01:00" to enable notifications, "00:00" to disable notification. I supposed that the Android Sample is already sending notifications but it was only for some special UUID_HEART_RATE_MEASUREMENT.

The code was replaced by:

for( BluetoothGattDescriptor descriptor : characteristic.getDescriptors() ) {
        descriptor.setValue(BluetoothGattDescriptor.ENABLE_NOTIFICATION_VALUE);
        mBluetoothGatt.writeDescriptor(descriptor);
}

And then I tried nearby handles, and it worked .

Remained 4 packets with 0x12 instead of 0x52. They belong to Bxx section where only PROPERTY_NOTIFY = 16 is set. Tried that and it worked too although the offsets were a bit different. Here is the diff for Bxx notifications.

And one small bonus — I was tired of watching the minidrone LEDs, so last night I started FreeFlight3 again and just spin the propellers . And there was a difference when compared to old the logged output:

02 40 20 18 00 14 00 04 00 52 43 00 04 01 00 04 01 00 32 30 31 34 2D 31 31 2D 31 31 00
02 40 20 1A 00 16 00 04 00 52 43 00 04 02 00 04 02 00 54 32 32 33 32 33 38 2B 30 31 30 30 00
02 40 20 0D 00 09 00 04 00 52 43 00 04 03 00 02 00 00

Guess, what is it? 0x30-0x39 are numbers … 2014-10-29 and T223238+0100 is better? So UUID 9a66fa0b-0800-9191-11e4-012d1540cb8e is Date/Time.

Now it is time to put it all together and replay it with 50ms sleeps (at least it looks like the messages are sent with 20Hz frequency).

p.s. I skipped bad news — I bought USB BT400 to enable Bluetooth 4.0 on my old laptop with Ubuntu Linux, fixed some issues with driver, but the drone refused to connect anyway …

13th November 2014 — The First Spin

Yesterday morning I finally managed to spin the propellers . Why it did not work sooner I am not 100% sure, but it could be due to timing (before I was sending messages without pause) and I had mistake in selected increasing byte, which was maybe in previous version too.

I decided to put even non-modified source on the github (diff), so if you want you should be able to repeat my steps. Please let me know if you encounter any problem.

I tried to record a demonstration video this morning, and sure enough it did not work. There is still a big TODO for version 0 (homologation, climb one step) and it is necessary to be able to stop the robot. There will be new application button for "Tour the Stairs", but it is not there at the moment, so what you need to do is enable notifications via B01 service, B0E characteristic — and that works:

14th November 2014 — Not there yet

This is the first attempt for version 0 Tour the Stairs code. And it is not working, yet. The drone moves slowly towards the step (beware, if you would like to repeat my attempt do not forget to attach wheels and configure the minidrone to use them), but then it fails to fly a little bit up. The step friction is too big. You can take off in free area but it is a bit scary/dangerous.

Note, that now two more bytes are correctly interpreted (see diff). Instead of 16bit value there are two 8bit values, turn right/left and up/down. Both values are probably percentage, i.e. 100 is maximum and you can use -100 for opposite direction.

15th November 2014 — Hints

I am no longer alone to fight this minidrone-stair-climbing task . And the other guy knows some great details.

For example, I can confirm that the last four bytes in AT*PCMD should be interpreted as a float number. I am still not absolutely sure what it does (I have to test it) — it is probably a multiple for other byte parameters? Do you remember me about writing that first 16bit number looks like forward speed (see forum)? Well, I was wrong. The same command is used also for flying and there would be missing parameter for tilt left/right. When you are rolling on the wheels you will not notice it (yeah, I am just trying to find an excuse ). Now I tried to replay all my old FreeFlight3 logs and I can confirm that both forward/backward and tilt left/right values are within range -100..100 (see fixed parse.py diff).

20th November 2014 — Takeoff and landing

You probably noticed that there was no „homologation video” posted till the deadline 17th November 2014 :-(. I overcome some problems like interpretation of the first two bytes in AT*PCMD command: when I send 8000 as 16bit integer it moved in reasonable speed forward, but when I set only byte part of it (hex(8000)='0x1f40', i.e. 0x40 and 0x1F) and the second byte zero then it did not move at all. The explanation was easy — it is not the first byte for the moving forward, but the second byte. And tilt byte helped to roll.

A friend of mine (PetrS) suggested to use nicer code via ByteArrayOutputStream and DataOutputStream but I did not properly solve endians for the float number (see diff).

So what is the problem? I switched to manual control and used FreeFlight3 and I did not manage to climb the step even with several attempts. Sigh. So it is hard to convince the machine to do it autonomously if I cannot do it manually. Note, that I was only driving on the floor and near the step I switched to full power up but it hardly lifted and it looked very dangerous.

My last chance is to fly to the upper step. So last night I pushed takeoff and a second later land. I was very very surprised how smoothly it went up and down when compared to older AR Drone 2.0. Again I parsed btsnoop_hci.log and I found there these new commands:

02 40 20 0D 00 09 00 04 00 52 43 00 04 05 02 00 01 00
…
02 40 20 0D 00 09 00 04 00 52 43 00 04 06 02 00 03 00
…
02 40 20 0D 00 09 00 04 00 52 43 00 04 07 02 00 03 00

The first is takeoff but I was not sure which one is landing. Note the 5th number from the right — it is again increasing like for AT*PCMD and 0x43 is actually handle for settings. See older post and compare this handle with setting date and time . So landing command was pressed twice.

Now it is early in the morning so I do not want to try autonomous takeoff/landing but I will probably publish the code, so if there is anybody interested you will know what I am talking about.

The plan is to takeoff to 20cm, fly forward and land. Unfortunately I did not find out how to collect data like altitude, absolute position, even if the takeoff/landing is complete … so it will be controlled only by time. At the moment it is probably „the simplest thing which could possibly work” .

p.s. sure enough it flew to 1 meter and then it refused to land … I am glad that the old trick that you take the drone by the frame (here wheels) and turn it up-side-down works

21th November 2014 — Emergency Stop

Last night I confirmed that it was not an accident that Rolling Spider did not want to land shortly after takeoff but „feature”. I tried to repeat landing during takeoff in FF3 (Free Flight 3), and it does nothing until you finish the takeoff sequence.

So there are two possibilities now:

climb the stairs with 1m high takeoff/land sequence
use Emergency Stop to cut all motors even during takeoff

The byte-sequence for Emergency Stop is:

02 40 20 0D 00 09 00 04 00 52 46 00 04 01 02 00 04 00
…
02 40 20 0D 00 09 00 04 00 52 46 00 04 02 02 00 04 00

So it is again another handle (0x46) and it has its own counter (5th byte from the end) — very similar to takeoff/land.

This code managed to „jump” on the step (the AR Drone 2.0 paper box), but it failed during video recording and now the battery needs recharging … so next test tomorrow.

23rd November 2014 — Battery packet

A few days ago I came across these inputs:

71.973 1: 02 40 20 0E 00 0A 00 04 00 1B BF 00 02 06 00 05 01 00 5F
72.108 1: 02 40 20 0E 00 0A 00 04 00 1B BF 00 02 07 00 05 01 00 5E
…
72.940 1: 02 40 20 0E 00 0A 00 04 00 1B BF 00 02 0E 00 05 01 00 57

The log was from experiment in FreeFlight3 with takeoff and I remembered that battery status was at the end 87%. The last number in the packet was decreasing by one and the last one 0x57 is 87 … so I think that now we have „battery packet”.

It was necessary to add more notifications (see code). The plan is to enable notification all available handles but time is getting short (the contest is on 29th November 2014). At least I finally recorded homologation video („jump” one step).

24th November 2014 — RC toy?! (5 days)

Bad news — it looks like that current version of Rolling Spider does not support sending of speed/position/altitude status :-(. The machine has to know it but it is not necessary for Free Flight 3 application. Experimentally I enabled all available notifications (see diff) and I have not see anything „interesting”.

So what can we do without any feedback? I tried takeoff with down command … crazy combination but it is still moving up anyway. What I would try is to takeoff, wait until it is flying and then move down without landing. For that we will need to know flying status.

With help it should be these messages:

0: 02 40 20 0D 00 09 00 04 00 52 43 00 04 05 02 00 01 00 = takeoff
1: 02 40 20 11 00 0D 00 04 00 1B BC 00 04 15 02 03 01 00 01 00 00 00 = takingoff

0: 02 40 20 0D 00 09 00 04 00 52 43 00 04 07 02 00 03 00 = land
1: 02 40 20 11 00 0D 00 04 00 1B BC 00 04 18 02 03 01 00 04 00 00 00 = landing
1: 02 40 20 11 00 0D 00 04 00 1B BC 00 04 19 02 03 01 00 00 00 00 00 = landed

0: 02 40 20 0D 00 09 00 04 00 52 46 00 04 01 02 00 04 00 = emergency cmd
1: 02 40 20 11 00 0D 00 04 00 1B BC 00 04 02 02 03 01 00 05 00 00 00 = emergency

25th November 2014 — Takeoff workaround (4 days)

Finally there was more serious testing today. I experimented with a takeoff workaround, i.e. complete takeoff sequence, return back on the ground with move down command and then move forward on the ground. The results were interesting:

sometimes notifications failed to turn on (i.e. the drone was infinitely waiting for takeoff sequence completion, or there were no data about low battery)
when the drone reached the ground with move down command it was oscillating back and forth for a while
as the battery was lower and lower the drone finished takeoff and switched to hovering even 20cm above the ground (instead of 1 meter)

The main problem was going „against the wall” without any feedback. Jessica just hit it hard and jumped back. This would mean falling down from the stairs. It is possible to limit vertical speed but the minimal value you can set is 0.5m/s. Maybe it is just limitation of Free Flight 3, and the speed could be set lower?? But the battery was dead I did not want to wait at school for 90 minutes until it would be fully charged.

p.s. one new byte sequence:

1: 02 40 20 11 00 0D 00 04 00 1B BC 00 04 04 02 03 01 00 02 00 00 00 = hovering status

26th November 2014 — Ver0 revised (3 days)

First of all I would like to thank icornerhightech.cz for lending me another drone (Jessica B./Blue/Backup) and extra battery. Now I should survive 8 rounds on Saturday with much lower risk (I wanted to write „without any problem” but we all know that something will „show up”).

Second, some issues I have seen yesterday were not caused by low battery but by re-starting the testing application. If I turn the minidrone off and on again it works usually fine.

Third, I wrote usually because once it did not work fine and I experienced very ugly crash. Even the wheels and one propeller fell off :-(. Maybe bad calibration at the start?? The drone just went in 20deg angle instead of vertical takeoff, hit the wall and fell on the ground. It still runs fine, so the mechanics is very robust!

I do not know if I mentioned battery and status info in the testing application. Now you can see if there are any updates coming, if the status is changing and how much battery is left.

Today we were testing in NTK Gallery and I had to tune ver0 — if there were 15 repetition Jessica jumped to 2nd step instead of 1st one. 12 repetitions = 0.6s was mostly just right, but when the battery was at 58% Jessica was not able to climb the step (sure enough in front of TV camera).

Because of the scary experience with one takeoff I will probably continue from ver0 (see function ver0ex(), where I called ver0 three times but I was not able to reach the 3rd step yet). So takeoff while moving forward and emergency stop is the way.

28th November 2014 — !!!STOP!!! (1 day)

Tonight I was preparing „contest version”. It was is not very great, but just a few changes to the last commit:

STOP button has to be separate — the drone is dangerous and it is very bad to guess it you switched it on or off
if you disconnect and connect again it works fine as if you reset the drone
the height of takeoff during first 12/20 to 18/20 seconds is quite random and probably depends on friction of the wheel with the step, battery status, … ???
Jessica Blue can be controlled with the same program but while the Red One flies almost 20cm the Blue One hardly took off 5cm … so I will use it as charger only.
I swapped approachStep with ver0 so there is at least a chance to score for the first step

30th November 2014 — Tour the Stairs

The competition Tour the Stairs is over. Jessica finished on 4th place (i.e. the last place) but I would still consider it success. In the best attempts she was able to jump/climb two steps, but emergency stop cutting the power to stop takeoff always disturbed it too much.

For the last two runs (straight and spiral staircase) the other team from Písek convinced me to try „prohibited” strategy with a rope. It was attached in the battery holder and it was approximately 1m long. I changed the code (see diff) to complete takeoff and to fly forward with power set to 10%. Now the hardest part was to enter the staircase (there were railings on both sides). Once it succeeded it was really superior to previous approach — I cut the first try on the staircase rest area but after the contest, just for a video documentation, Jessica was able to climb the whole staircase .

So there is another difference between old AR Drone 2.0 and Rolling Spider (I think that this change was also between old AR Drone and the next generation). As I said I was flying forward only but I never hit the steps. So the altitude is only relative and not absolute to the ground as for AR Drone 2.0. On the other hand it could be caused by sensor confusion because of the rope (bad sonar or camera readings?) so I am not sure.

What next? Well, I hope I receive some videos of the contest (the official one will be in January, but something else could be next week). And that is going to be end of this blog (both drones will be returned this Tuesday).

p.s. I still hope that Parrot will revise the firmware to send drone data (time, speed, altitude, reading from sonar and camera, estimate of position) and that it will be possible to stream some pictures over Bluetooth (now you can save them and then transfer them but I did not try that during flight yet).

What is the FOV of the camera?

4th December 2014 — Parrot released ARDroneSDK3!

Today I received two interesting e-mails related to Rolling Spider. The first one pointed me to another github repository … so somebody else was trying to modify Android BluetoothLE example like me .

The second mail trump the first one:

!!! PARROT RELEASED ARDroneSDK3 !!!

It is probably not publicly announced yet (?), but the github is ready. If you are lost in these tons of source code I would recommend to start from ARDrone3_commands.xml, where are all commands shortly described.

contact form

Results

2014-09-23T00:00:00Z

Robotour 2014 in a nutshell

It rained this year. And it was not just a few drops — it was a serious pelting rain, which would some robots easily flood out. So that's probably the first unforgivable memory to this year Robotour in Pilsen.

Do you see the path right down?!

Another positive memory is that all 12+1 teams arrived (well, the new team Blade XXII mask that till Saturday). There were five new teams and all fightet bravely, it is true that some with their own robots and software, but … AmBot finished „winner box” and big chances had also Polish team TAPAS Team, which was favorite in first runs (they did not start in 3rd pelting rain run, which was wise decision (*)).

The score was very bad this year. The paths were narrower than in Lodz and some were a grass ones. The top is the path on the right picture, which is mapped in OSM. So it was harder to judge and there were debates of some not perfectly mapped pieces of road. Also the 3rd rainy round was a source of some disappointment.

Although it was quite demanding, I am happy for Robotour 2014 — by the way is not the reason why we are doing that?

A few quick pictures

Safety tent

Naked E-liška

The half of Cogito

MarS

Plecharts

Warning

Overall score

Order	Team	1st run	2nd run	3rd run	4th run	Total
1st	Smelý Zajko	46	40	64	80	230
2nd-3rd	Radioklub Písek	0	0	2	179	181
2nd-3rd	AmBot	0	16	127	37	180
4th-5th	TAPAS Team	22	53	-	63	138
4th-5th	ARBot	11	0	49	73	133
6th	Cogito	0	16	-	13	29
7th	NDTeam	-	-	2	7	9
8th	JECC	2	0	0	0	2
9th-12th	AutoLUT2	-	0	-	-	0
9th-12th	Istrobotics	-	0	-	-	0
9th-12th	MarS	0	0	-	-	0
9th-12th	Plecharts	0	0	0	0	0

The winner is after five years of participation Smelý Zajko! Congratulation, also for endurance and patience.

Workshop

There were only Czech and Slovak teams on Saturday workshop, so after a while it was easier to talk in our native languages. I was relatively tired from Saturday, but the workshop gave me a plenty energy and optimism.

At the beginning Smelý Zajko described his hard way to the victory (they participated since 2010).

Radioklub Písek presented a story of burned motors of E-liška and „current magic” on new controllers.

Cogito finished their presentation with comment that is is not necessary to a build autonomous car and that the goals of Robotour can be reached by autonomous bike, cart or skateboard. See the last page of his presentation.

[The objective of the Robotour contest is to encourage development of robots capable of transporting you to work in the morning or to deliver the building material you have just purchased in an online shop. ]

ARBot showed his first results from stereo vision computed on gate array.

NDTeam, working in picoseconds timing business, presented his own robot with aluminium construction controlled by a 4-core computer of size of credit card.

Istrobotics mentioned synchronization of web camera PS3 (?) … you just have to solder proper inputs, but there are some issues with Windows drivers fro multiple cameras etc.

Simply all experts! The presentation often started like in alcohol/drog abuse therapy: „I am on Robotour since XYZ ...” Well, it is maybe addictive, so be beware!

Plans for Robotour 2015

Just brief points:

the Friday homologation will not be mandatory (it was already this year)
the order of teams in second and next round will be reversed, i.e. teams with high score will start at the end of the starting area
the payload remains mandatory
the runs are in all weather conditions

The next year is anniversary so we are thinking how to properly celebrate it. The preference for next location is still Czech Republic, but there are other alternatives like Großer Garten (Dresden). Smelý Zajko took some pictures on the way home in park in Hluboká nad Vltavou and comment it If Pilsen was difficulty level 4, the Hluboka is level 7! … so something to look forward.

Acknowledgement

I would like to thank especially to Petr Weissar and his student for creating the base. Without the tent we wound not survive the Saturday afternoon. Thanks to Westernbohemia university for support: the dormitories next to the park were very handy and the conference room in Academic center with café íčko in city center was a nice alternative for workshop.

If you find this contest interesting and you would like to participate next year, or you have interesting pictures or videos, please let us know via contact form.

(*) I am reading conclusions form last year Poland and one was that a team can give up one attempt without punishment. On the other hand TAPAS Team would score similarly only Smelý Zajko would have lower score and the order would be identical.

Photos

authors: Zdeněk Kubík and Petr Weissar

The Cup

AmBot

ARBot

AutoLUT2

Istrobotics

JECC

TAPAS Team

AmBot

ARBot

ARBot support

Common beer

Cogito

JECC

MarS

NDTeam

Plecharts

Radioklub Písek

Rain

Robots

Smelý Zajko

Start 2

Start 3

Start 4

TAPAS Team

Stan

E-liška and ZCU

Wet coordinates

Robo-art

Introduction of teams

2014-08-12T00:00:00Z

Teams

YouTube playlist of all registered teams

AmBot (CZ)

youtube http://youtu.be/RC306Ex3aVU

Robot Ferda is modified kids electric car ("ride-on") for Robotour 2014. The main control system is Arduino based with ATmega2560. It takes care of motors control, integrates magnetometer and two sonars for obstacle detection. It also reads data from external GPS receiver via Bluetooth converter. Arduino software provides possibility to define GPS waypoints and the car tries to navigate by them. It can accept also commands from Bluetooth converter. The goal is to extend system with Android smartphone running simple application with visual navigation (to keep the robot on the road).

ARBot (CZ)

youtube http://youtu.be/ve-CoNReFTw

ARBot is a small robotic vehicle constructed for outdoor competitions of autonomous robots. The robot has four-wheels chassis, each wheel is powered and has encoder. Robot has camera, GPS, AHRS unit and three sonars. The computation is handled by DSP BF537 with power 1000 MIPS.

AutoLUT2 (PL)

youtube https://www.youtube.com/watch?v=LBPyjKe679Y

The vehicle is driven by 36V DC motor and it can turn using an electric ram. It is controlled by the arduino microcontroller and additional computer.

Blade XXII (SK)

youtube https://www.youtube.com/watch?v=BnWsfl_jUJA

Leopard Pro 36 converted to eletric power STM32F103 - motor/servo control, sensors Radxa Rock - navigation, optical recognition

Cogito (CH/CZ)

youtube http://youtu.be/2H_66EUvTKg

Lot of hardware, lot of software, lot of fun.

All electronic of robot B-trix is, as in 2012, attached to electrochassis 1:5 and low-level control handles Arduino Duemilanove. There was serious upgrade of sensoric part - new laser rangefinder, compass has inclination compensation, number of sonar is tripled, "ordinary" camera was replaced by stereo camera. Xtion remained but nobody expects anything from it. GPS, magnetic encoder, gyroscopes and accelerometers are common sensors on this contest.

The high level control is managed by mini-ITX with Atom processor There are so many Ethernet toys that robot carries its own intranet. Software is mixture of Python, C++, C and bash. A plenty of vision, plenty of planning, but it is almost impossible to compute it in time.

Istrobotics (SK)

youtube http://youtu.be/395K47_SHfc

The base of the robot is modified RC model TRAXXAS E-MAXX (3903). It is equipped with webcam, GPS, sonars HC-SR04, IMU with 3D compass and magnetic IRC. The basic sensors handles arduino mega. The image processing and GPS runs on 8" tablet with Intel Atom and Windows 8. The program is written in C++ and is using OpenCV.

JECC (DE)

youtube http://youtu.be/Doo27P37gCs

4 wheel drive powered by Lipo 42 5000 mAh motorcontroller: DRV8800 Controller: Beagle Bone Black

NDTeam (CZ)

youtube http://youtu.be/FWOTWIGRihw

Robot Robík is own construction inspired by robot Orpheus. It weights approximately 15kg, driven by two DC motors with planetary gearbox. The wheels are connected with toothed belt. The control is own electronics based on ARM processor Cortex M3. Equipment: GPS + 9 DOF AHRS, sonar, camera+OpenCV on platform Odroid U3 for road detection.

Plecharts (CZ)

youtube http://youtu.be/j4cK0xIAdUU

The robot construction is from freely available parts. The skeleton is mounted from aluminium profiles. The drive is by two electromotors with maximal combined power about 2.6 kW. The power supply is provided by two Pb accumulators 12V 72 Ah each (usually only one is used). The maximal speed is about 0.5 m/s. All modules (sensors, motor control, etc.) communicate by TCP/IP protocol.

Software is written in pure C++, and runs on older notebook. The road recognition uses neural networks. Park navigation may use GPS, maybe even some map.

Radioklub Písek (CZ)

youtube http://youtu.be/bAk96kyaTgc

Radioklub Písek is participating already for the 6th year on robotic outdoor competitions. Last year we completed new robot E-liška, and we took it for the first time to ROBOTOUR, where we reached 3rd place, again. And this year 3rd place on Robotem Rovně and 3rd place on RoboOrienteering . After small upgrades we count with it on ROBOTOUR 2014. E-liška dimensions are 95x60x48 cm and weight approximately 40 kg. The on-board voltage is 24V, provided by two gel accumulators 12V/18Ah. E-liška has spring-loaded four-wheel chassis with Ackermann steering and all wheels are powered. Each wheel has its own control unit. We use Lidar Sick , GPS a 9dof unit for orientation. The main control handles notebook, and motors have its own module with STM32. The power control is handled by H-bridges of own construction. The main program is written in Python and runs on Linux.

Smely Zajko (SK)

youtube https://www.youtube.com/watch?v=dAUn9LSuKm0

HARDWARE: Parallax (Motor Mount and Wheel Kit), encoders, 2xHB25 Sbot board (AVR ATmega128, designed and assembled by David Gustafik) PC ASUS UL30V 5x SRF-08 GPS NaviLock NL-302U USB SiRF III Compass with tilt compensation (HMC6343) AVR ATmega8 (compass driver) Camcorder Panasonic SDR-T50 (or USB webcam) video grabber EasyCap DC60 USB 2.0 TV DVD VHS Video Adapter W / Audio AV Capture TV DVD CVBS-Adapter usual usb hub Power: HAZE HZS 12V 9Ah handmade wood & aluminium base (contributions by Miroslav Nadhajský and Pavel Petrovič) red power switch, and power circuitry (contributions by Richard Balogh)

SOFTWARE: Ubuntu 14.04 Desktop LTS Netbeans OpenCV Smelý zajko controller utilizing an Artificial Neural Network (FANN) (result of Miroslav Nadhajský master thesis) SBOT firmware written in C/AVR Studio (David Gustafik) with modifications (Pavel Petrovič) Compass driver with serial port interface written in C/AVR Studio available at https://code.google.com/p/smely-zajko/

TAPAS Team (PL)

youtube https://www.youtube.com/watch?v=RYOczhVBujE

Robot's hardware was intended to be simple, modular reliable. It's main body is built from aluminium profiles. Motors are rigidly mounted with wheel mounted on their's shaft. Considering electronics, we use prebuild motor drivers, Discovery STM32F4 evaluation board with cape made by us, AHRS sensor, Hokuyo laser scanner, GPS receiver and nettop computer. All peripherials are conneted by USB link. Some parts, especially fixings, were 3D printed.

Software is composed of 3 main modules: localization, movement constraints and navigation. First one uses Extended Kalman Filter and data from AHRS, encoders and GPS to obtain global position. Second one uses Hokuyo laser scanner and camera to compute movement constraints. It consists two parts: obstacles detection from point cloud (agregated from laser scans) and terrain classification from combined camera image and point cloud intensity values. The last module uses Vector Field Histogram for local planning and A* for global one. As a base we use Ubuntu operating system. For telemetry and development purposes we use remote desktop and dedicated GUI.

If you would like to somehow support this contest or you have some comments/question, please use our standard contact form.

Robot Challenge 2014

2014-02-10T00:00:00Z

Note, that this is not just „English translation” of Czech article. It seems that fund raising campaign failed (well, there is still a week to go, but I doubt it will jump to required limit), so I decided to „log” my progress at least in English. Let me know, if you find this interesting .

AirRace rules (PDF file)

We are playing with two robots" Heidi and Isabele. Both are AR Drone 2.0 from Parrot.

The source code is available at github: https://github.com/robotika/heidi

Here are some pictures from older Czech blog:

Navigation line

Scary landing area

Camera bottom view

Blog

February 10th, 2014 - cv2

Yesterday I was fighting with OpenCV for Python, which seems to be just perfect tool to help us reach the goal. My problem was that I was following only the Python setup and it kind of worked (loaded and displayed image, for example). But as soon as I need to to parse video with H264 codec I was doomed :-(. There are many comments on the web that integration with ffmpeg is complicated, but there were already some DLLs prepared?!

Well, next time start with basic OpenCV installation and then the Python part. The only critical detail is that you have to set system path to binaries, ffmpeg DLLs in particular … yeah, quite obvious, I know.

Here is the first image dummy processing, and here is video replay code recorded by Isabele.

February 10th, 2014 - PaVE and cv2.VideoCapture

cv2.VideoCapture is nice function for access to video stream. Thanks to integrated FFMPEG you can read videos with H264 codec, and this is something used by Parrot AR Drone 2.0. The bad news is that it is not clean H264 stream, but there are also PaVE (Parrot Video Encapsulation) headers.

Does it matter? Well, it does. Yesterday I have seen couple time crashing Linux implementation, and on Windows you get several warnings like:

[h264 @ 03adc9a0] slice type too large (32) at 0 45
[h264 @ 03adc9a0] decode_slice_header error

The reason is that H264 control bytes 00 00 00 01 are quite common in PaVE header and then the decoeer is confused. Note, that most projects ignore this:

It is OK, if you just need „some” images, but once the decoder is confused, you loose up to 1 second of video stream until I-frame is successfully found.

So how to work around it? I had two ideas, both crazy and ugly. One is to create „TCP Proxy”, which would read images from AR Drone 2.0 and offer them to OpenCV via TCP socket. Another possibility is temporary file, where ret value is sometime False, but you can replay the video in parallel.

cap = cv2.VideoCapture( filename )
while(cap.isOpened()):
    ret, frame = cap.read()

There are probably more complex solutions like directly control FFMPEG API, but it is pity to lose this otherwise nice and clean code . Any ideas?

February 18th, 2014 - I-Frames only

Well, cv2 is still winning. I tried both — temporary file as well as TCP Proxy. It took a bit too long to start simple server (several seconds), which is not acceptable in „real-time video processing”. Temporary file looked more promising, but on Linux it was completely unusable. Hacking around cv2.VideoCapture is maybe not the best idea, and proper solution with buffer frame by frame decoding would be the right way, but …

Today's fall back was usage of I-Frames only. It does not work if you create video from I-Frames only, but you can create new file, write there single I-Frame, and then read it with cv2.VideoCapture. I know crazy. But it works for now and hopefully in meantime we will find some more effective solution. For details see source diff.

Heidi has now new flying code airrace_drone.py. It is glued together not to interfere with drone Isabele. Simple processFrame is then in airrace.py. Adaptive thresholding, detection of contours and for reasonable size contours compute bounding rectangle (with rotation).

detected rectangles

February 19th, 2014 - Rotated rectangles

Writing a robo-blog in English is not as much fun as in Czech. At first my English is much worse and at second I do not have a clue if there is anybody reading it. That's something Fandorama was good for. I've got clear feedback how many people are interested so much that they would even support it. And almost nobody was interested in this blog in Czech, so … you get the picture. If you find a bit of information useful for you let me know. It helps to brighten up mine otherwise depressed spirit .

What I want to do now is revision of logged data from yesterday. The cv2 logs are normal text files with results of image analysis + number of updates between them. The last program was supposed to turn drone based on third parameter of first rectangle. Just a quick hack that I see that there is some feedback going on and that I can replay already recorded data with exact match.

NAVI-ON
-26.5650501251
-15.945394516
-45.0
-88.9583740234
NAVI-OFF

This is the output in console — angles used for rotation. It is also my first TODO, to add there video timestamps.

182
[((817.0, 597.0), (10.733125686645508, 12.521980285644531), -26.56505012512207),
 ((223.17218017578125,
 440.579833984375), (319.85919189453125, 60.09449768066406), -88.75463104248047)]
198
[((298.0, 508.0), (58.51559829711914, 319.50067138671875), -15.945394515991211)]
198
[((982.2500610351562, 492.25), (29.698482513427734, 17.677669525146484), -45.0),
 ((951.9046020507812, 
507.705078125), (60.70906448364258, 84.2083511352539), -69.02650451660156)]
185
[((741.7918090820312, 247.95079040527344), (43.32011032104492, 72.9879379272461),
 -88.9583740234375),
 ((1035.0001220703125, 14.999971389770508), (16.970561981201172, 
12.727922439575195), -45.0)]
199

My second TODO would be to round/truncate floating numbers to integers. It would be easier to read and remember, my goal is to turn left/right, i.e. something if mass center is bigger or smaller than width/2 …

182
[((817, 597), (10, 12), -26), ((223, 440), (319, 60), -88)]
198
[((298, 508), (58, 319), -15)]
198
[((982, 492), (29, 17), -45), ((951, 507), (60, 84), -69)]
185
[((741, 247), (43, 72), -88), ((1035, 14), (16, 12), -45)]
199

Better? I think so . Now I would almost guess that it is ((x,y,(width,height),rotation), but API is so simple that it is worth it to test it:

import cv2
import numpy as np
frame = np.zeros( (1200,720), np.uint8)
rect = ((223, 440), (319, 60), -88)
box = cv2.cv.BoxPoints(rect)
box = np.int0(box)
cv2.drawContours( frame,[box],0,(0,0,255),2)
cv2.imshow('image', frame)
cv2.imwrite( "tmp.jpg", frame )
cv2.waitKey(0)
cv2.destroyAllWindows()

Well, you probably know how to make it simper, but this is whole working program, which will display image and save it as reference. The width/height of image is swapped and I do not see there anything :-(, so it is not that great. In general I would recommend OpenCV-Python Tutorials. Note, that they are for non-existing (or better to say for not so easily available) version 3.0, and some cv2.cv.* functions are renamed (see notes).

So how to fix the program? Back to Drawing Functions example. Great!

# Create a black image
img = np.zeros((512,512,3), np.uint8)

so my fix is

frame = np.zeros( (720,1200,3), np.uint8)

and now I see red rectangle. With modification rect = ((223+300, 440), (319, 60), -88) and rect = ((223+600, 440), (319, 60), 0) I see that first coordinate is really (x,y), then (width,height) and finally angle in degrees, where Y-coordinate is pointing down … maybe I was too fast going to conclusion. One more test rect = ((223+600, 440), (319, 60), 45), so (x,y) is the center of rectangle! And now is time to read the manual . Hmm … The function used is cv2.minAreaRect(). It returns a Box2D structure which contains following detals - ( top-left corner(x,y), (width, height), angle of rotation ). Test first! Here is my accumulated image:

Drawing rectangles

So it was not that boring at the end . No wonder drone did what she did. First it navigated to random square, then to perpendicular angle (but be aware … width is not necessary bigger than height, see ((298, 508), (58, 319), -15) … homework for you, was for this particular box correct to navigate to angle 0?

February 21st, 2014 - The Plan

Aleš wrote me e-mail, that it is not clear (from my description) what is the exact plan. And he is right!

The plan is to use standard notebook (one is running Win7 OS for Heidi, another Linux for Isabele) where all image processing will be handled. Navigation data plus video stream from drone is received via WiFi connection. WiFi is also used for sending AT commands to drone. The code is written in Python. For image processing is used OpenCV. The code is publicly available at https://github.com/robotika/heidi.

Aleš is also right regarding task complexity. According to his estimates it will be necessary to flight with average speed at least 1.5m/s. With video sampling rate 1Hz (current status), that will be quite hard. It is a challenge … like what the name of competition says „Robot Challenge”! See you in the area!

p.s. if this blog was sufficient impulse for Aleš to install and start to play with „OpenCV for Python” then I think that at least one alternative goal was already reached

February 26th, 2014 - Version 0 (almost)

Version 0 should be the simplest program solving given task. In the context of AirRace this means complete at least one 8-loop. And we are not there yet. Camera integration always slows down the development …

In meantime I also checked the other competitors. There are (or yesterday were) 12 of them! So it is clear that while in 2012 one completed round would be enough to win the competition, in 2013 it was enough for 3rd place (see Heidi) and in 2014? We will see.

Here is the source code diff for test-14-2-25. There were some surprises, as usual .

Z-coordinate

Probably the biggest surprise was test recorded in meta_140225_183844.log. Drone aggressively started and reached 2 meters height instead of usual 1 meter. That would be still OK, and soon it would reduce the height to desired 1.5 meter, but according to navdata the height was 1.43m! Moreover the restriction for minimum sizes of detected rectangles was set to 200 pixels so no navigation data was used/recognized.

Rectangles

One more comment about rectangles detection. Originally there was no limit on the size of rectangle, and that was wrong. Here are some numbers so you get an idea: at 1m the 30cm long strip is detected as 300 pixels rectangle. This means that at 2m it should be approximately 150 pixels (current filter value). Thanks to Jakub bad rectangles are now visible in the debug the output:

Blue = rejected rectangles

Now I am browsing through recorded videos and looking for „false positive” or how is it called — rectangles which should not be selected and were selected, and … here is a bad surprise:

Mixed video frame

We are still processing recorded video which should be complete. So it is possible that PaVE packet contained only part of the video frame. Another TODO.

Corrections in Y-coordinate

It turned out that it is necessary to make also sideways corrections. Sure enough there was a mistake in sign: image x-coordinate is drone Y coordinate and it should be positive for x=0. Fixed.

Hovering

The most important step was insertion of hovering. Just to get the drone at least a little bit under control. The video frames are processed at 1Hz so by setting the proportion between 0.0-1.0 splits the time for steering and hovering. 0.5s was too slow 0.8s too fast and 0.7 was reasonable at this stage. The plan is to trigger hovering only when the error grows above some predefined limit.

Results 14-2-25

The drone is extremely slow at the moment. In one minute it was not able to complete a semi-circle. It will be necessary to recognize which part of 8-loop is it and steer instead of go forward as default command.

Another detail to think about is going backward instead of forward. The down pointing camera is shifted little bit back. It may simplify navigation in some cases. Like test logged as meta_140225_185421.log, when Isabele flew across our table (we do not have protective nets), because only in one frame it had seen perpendicular strip … and then it was too late .

February 28th, 2014 - Bug in log process

Today I was investigating timestamps of video vs. navigation data. What scared me a lot was five seconds gap between initial frames:

1548.280426
0
1549.273254
-16
1550.32669 <— !!!!!!!!
-18
1555.131744 <— !!!!!!!!
-9
1557.108245 <— !!!!!!!!
-14

… here you see time in seconds and detected angle in degrees, just to find/match corresponding frame. So what is the problem? Here is the log file for image processing:

…
[((1009, 643), (74, 48), -18), ((1013, 624), (16, 12), -45), ((464, 482), (46, 266), -7), …]
178
[]
750
[((172, 209), (15, 8), -9)]
188
[]
184
[((327, 551), (43, 239), -14), ((244, 235), (43, 216), -14)]
190
[((1045, 676), (31, 38), -45), ((268, 483), (43, 236), -16)]
199

The results are interlaced with the number of updates of drone internal loop (running at 200Hz). The problematic number is 750 … so for such a long time there was no result?! This would mean that drone navigation was based on video frame several seconds old!

Well, good news is that in reality that was not so bad and the mistake is in logging. It is necessary to have all results different in order to store changes. Here for several seconds were no rectangles detected (i.e. empty list []) and these results were packed into one. If there would be timestamps and frame number the results would be always different and then it should work fine.

It is fixed now, and ready for new test (see github diff).

March 3rd, 2014 - Z-scaling

A quick observation based on today's Jakub tests: it is about strip (x,y) coordinates scaling based on the drone height (Z-scaling). The distance is estimated individually for each strip. I could not see any reason why three strips in line:

strips in line

are not linear in the log viewer:

pink centers of strips

?! I already told you the reason … the z coordinate is estimated separately for each strip (based on its length) and the middle is much shorter (due to a light reflection), so it is expected to be further away …

March 4th, 2014 - A Brief History of Time

I could not resist to use Stephen Hawking book title, sorry . AR Drone 2.0 has at least two time sequences: NavData and Video. They are encoded differently (see for example Parrot forum), but good news is that they are probably related to same beginning of time. In this case not Bing Bang, but only Drone Boot.

The video stream is delayed. Not much for recording but maybe a lot for fast drone navigation. The number vary around 0.3s, but you can see also values like 0.44s and probably more. Here you can see the difference if you tight video frame to current pose or if you keep poseHistory:

before time correction

after time correction

March 5th, 2014 - test-14-3-4

Yesterday tests were quite discouraging :(. We had visitors and the best we could do was finish 1/2 of 8-loop. After the second turn Isabelle decided to „turn right” on the crossing and that was it. So we still do not have working version 0!!!

Now it is time to go through the log files and look for anomalies. Starting from the last test I can see 90 degrees jump in heading. This is probably normal as internal compass is taken into account and it takes while before enough data are collected.

Isabelle is not very high (approx. 1m) at the start position. It is pity that it won't recognize partially visible strip. Moreover the recognized strip is, due to light reflection, shorter so distance estimate is not very accurate.

There is an obvious problem with Y-coordinate correction. It is „one shot” only, with dead zone 5cm. For 10cm error it creates beautiful oscillations . On the other hand correction in angle with dead-zone +/- 15 degrees (!) does not correct for first three image snapshots at all … OK, now it is clear. Who programmed that?!

Another source of problems is failure to detect transition from circle to line. As result circle corrections are used instead of line navigation.

There are only 3 more tries/test evenings to go, so we will have to pick from the following list:

switch to smaller images, stream preview video (i.e not recorded), faster frame rate
C/Python binding to FFMPEG and parsing intermediate images (i.e. more than 1Hz video update rate)
MCL (Monte-Carlo Localisation) based on strips position
improve quality of strips image recognition
big image of the floor from matched and joined mosaic images
rectangle merging (detected two small which are split due to light reflection)
position driven corrections

March 6th, 2014 - Cemetery of Bugs

Hopefully I can call with this title today's commit . Sometimes it is hard to believe how many bugs can be in few lines of code. I already mentioned couple yesterday, so few more today:

drone.moveXYZA( sx, -sy/4., 0, 0 )

This was supposed to be replacement for hover() function in Y coordinate. The idea was that you send the drone with the opposite speed then which is flying now. So what is the problem? sy is not the drone speed! It is the command after image analysis for tilt right or left. It should be drone.vy instead.

Another bug, probably not so critical, is a block of code with condition, if any strip was detected. If there was no strip then history of poses was constantly increasing (in proper way it is cut with the timestamp of the last decoded video frame). Solved via completely removing the if statement — it is no longer needed. I no longer worry that I do not see any rectangle.

Then there was a surprise with ARDrone2.vy speed itself. I never need it, except for drone pose computation, and so it was with signs as you can get them from Parrot. Fixed in ardrone2.py.

So what next? Hopefully today will Jakub test new code with reference circles. Instead of navigation based on old images, Isabelle should now use circular path estimated from the image. I should be much better, but will be?

March 10th, 2014 - Shadow

Jakub sent me results of Friday tests with navigation along reference line with comment: we've got problems with shadows … and he was absolutely right:

Shadow --- video_rec_140307_150703.bin:630

This is not the first time when

ret, binary = cv2.threshold( gray, 0, 255, cv2.THRESH_OTSU )

fails. On the other hand it is not surprise — there are three peaks in histogram instead of two. The worst thing on this particular picture is that it not only does not detect any of three black strips, but it evaluates garbage as perpendicular strip!

For quick test I used only Irfan histogram to show what I mean:

Image histogram

Note, that I ignored red/green/blue pixels and picked blue histogram, because there you can see the best three peaks.

March 12th, 2014 - Finally Version 0!

I can sleep again . Not much (it is 5am now), but still it is much better . Yesterday Heidi finally broke the spell and after several weeks completed the figure eight loop!

Are you asking why Heidi and not here sister Isabelle? Well, there was a surprise. Isabelle does not have working compass! Yesterday I wanted to add extra direction filtering, basically to make sure that I am following the proper line. So I integrated magneto tag into ARDrone2 variables. I was using last 2 minutes test from Jakub (OT that was maybe another moment for celebration when Isabelle was able to autonomously fly and navigate for 2 minutes) and to my surprise there were constant raw data readings from compass:

compass:	
-55	-57	-191	
193.7109375	187.55859375	671.484375	
-120.508392334	108.018455505	671.484375
314.219329834	79.5401382446	0.0	-121.496063232
0.0	-137.727676392 0x01	2	
402.916381836	-241.082290649	0.0546875

This corresponds to following named variables:

values = dict(zip(['mx', 'my', 'mz', 
  'magneto_raw_x', 'magneto_raw_y', 'magneto_raw_z', 
  'magneto_rectified_x', 'magneto_rectified_y', 'magneto_rectified_z',
  'magneto_offset_x', 'magneto_offset_y', 'magneto_offset_z',
  'heading_unwrapped', 'heading_gyro_unwrapped', 'heading_fusion_unwrapped', 
  'magneto_calibration_ok', 'magneto_state', 'magneto_radius',
  'error_mean', 'error_var'], magneto))

This was kind of magic — it is almost hard to believe how data fusion of gyro + compass perfectly hide this problem. In particular if you watch variables heading_unwrapped, heading_gyro_unwrapped, heading_fusion_unwrapped then first (compass heading) was constant (as well as mx,my,mz and others). Second gyro heading was slowly changing and finally in fusion heading was originally winner compass, but later on it was loosing against gyro . It is really hard to believe that it worked so well and except longer flights you would not notice it.

I could not resist and tried immediately Heidi at home in the kitchen. I cut the motors as soon as they started to turn (it was 6am, so neighbours would probably complain). There were two test: first drone pointing south and then drone pointing north. Values were different, so it is issue of Isabelle.

Now, what is worse than not working sensor? Any idea? Well, there is even worse case — sometimes not working sensor! After several tests flights with Heidi yesterday we tried final working algorithm also with Isabelle. And it worked fine! Do you see the persistent direction of 8 figure?

Isabelle with working compass

Just to get an idea, visible only form raw data, here is older 2 minutes flight mentioned sooner:

Isabelle with broken compass

Note, that I had to change the scale so you will see the crazy trajectory …

Fun, isn't it? Now I would guess that communication with compass module sometimes fails and the only possibility is to power off the drone and restart the whole system. Maybe. Another TODO to go through older log files and check for this behavior.

So that was the bad news … now the good ones … and the title of today's post. We finally managed to complete the 8-loop and repeat it several times. There were three changes (see older diff):

decreased speed from 0.5m/s to 0.3m/s
wait on start if no strip was detected
fixed bug — missing abs()

The speed is obvious and does not need any explanation. The second point is related to elimination of most bad starts — the takeoff sequence is completed at about 1m above the ground and reasonable view from bottom camera is at about 1.5m. The algorithm slowly increases the height, but it should not leave the start area.

And the third one is the funny one . That was the reason why Isabelle or Heidi took wrong turn on the crossing. It was quite repeatable … fixed.

p.s. I wanted to post also link to YouTube video, but my video converter no longer works … it looks like there were some changes in FFmpeg … but maybe it is only to change avcodec_get_context_defaults2 to avcodec_get_context_defaults3 and replace AVMEDIA_TYPE_VIDEO by NULL as in this example. It looks OK and here is the video … note the strange behaviour near the end. Still plenty of work to do.

March 13th, 2014 - Frame contour

Did you watch the video of the first Isabelle's successful flight to the end? Can you guess what happened? If you have no idea then frame number 3630 from video_rec_140311_192140.bin can help you:

False strip on the frame contour

And here is the report of changes near the end:

FRAME 3540 I I
FRAME 3570 I I
TRANS I -> X
FRAME 3600 X I
TRANS I -> L
FRAME 3630 L L
NO dist change! -0.989344682643
TRANS L -> X
FRAME 3660 X I
FRAME 3690 I I
FRAME 3720 ? I
FRAME 3750 ? I
TRANS I -> L
FRAME 3780 L L
FRAME 3810 L L
FRAME 3840 L L
FRAME 3870 L L
NAVI-OFF 120.001953

Isabelle wrongly interpreted this image as turn left (!) … well, yeah … one pair looks like two following strips on left turn circle. And then there is a transition from circle to crossing, which should not ever happen where drone picked wrong line for navigation and started to fly loops in the opposite direction.

p.s. note, that Heidi (older sister with several injuries from the last year) in reality finished the 8-loop sooner, but in her last attempt (meta_140311_191029.log) unexpectedly landed in the middle of the test. What happened there? Low battery (on the start 39% and after landing 0%) … OK, so now we know that if battery is low done will automatically land .

March 14th, 2014 - cv2.contourArea(cnt, oriented=True)

VlastimilD wrote: the function cv2.contourArea has parameter "oriented", which is by default False and if set to True, it returns negative area which means the contour has the opposite orientation and that should be white on black i this case. So if you don't want those, setting oriented to True should filter them out in the following "if". … and he is absolutely right! Thanks for hint .

For details see OpenCV documentation.

p.s. I am sorry for misspeled name in the commit.

March 16th, 2014 - MP4_360P_H264_360P_CODEC

We have decided to switch to video recording in MP4_360P_H264_360P_CODEC mode. The previous mode was MP4_360P_H264_720P_CODEC, i.e. with higher resolution (1280x720) at 30fps. Our main motivation was to get rid of split images — cases, where you get mixed old and new image (for details see mixed video frame).

Another motivation was to reduce WiFi traffic and the size of recorded reference videos. So far so good. And yes, there was at least one surprise . The 720P video was not only with higher resolution but also on 30fps, and 360P video is running at 15fps! For us it is actually good news, because expected size will not be 4 times smaller but 8 times (in reality it won't be so big difference, because large and more frequent pictures are easier to pack).

March 17th, 2014 - Scale Blindness

Did you experience the feeling when the mistake is so big, out of range, that you won't see it? Here is the story: low resolution video does not work. The navigation algorithm is much much worse. Why? The error is too big to be seen … and it is already presented on all previous images .

Just for comparison here is one not working 360P image:

Example of 360P image

Do you see now, why cv2.THRESH_OTSU used to work fine before, but it is not working any more? No?

boring, right? The bottom facing camera has QVGA (320*240) resolution, according to Parrot specification . That resolution is neither compatible with 1280x720 nor with 640x360. But what is maybe not expected is that for high resolution you scale it 3 times (720=240*3) and fill the margin with black pixels, while for 640x360 you scale it twice (640=320*2) and cut the overlapping pixels! Histogram for high resolution video contained (1280-960)*720 totally black pixels …

March 18th, 2014 - Version 1 and MSER

It is still quite depressing — we would like to get 30 rounds and at the moment we are happy if we complete 3! Well, it is called progress, but up-to-now Version 0 with 8 times 8-loop was still better.

What is the main difference between Version 0 and Version 1? First of all it is the video input with smaller resolution. After yesterday discovery we know that the image is covering only 3/4th of reality visible on high resolution video. This means that you see two complete strips less frequently than before. There is a new class StripsLocalisation, which should compensate for this by taking into account also previous image and the movement of drone between snapshots.

There is not much time left — only one week so almost zero chance that we will integrate Version 2, which would handle absolute localization and compensate for multiple failures … oh well.

There was a good news at the end too: it is possible to use MSER (Minimally Stable External Regions) implementation in OpenCV in real-time image processing. Isabelle was able to complete several rounds with this algorithm. The video delay was comparable to normal binary threshold set for 5% of all pixels count.

March 19th, 2014 - MSER and negative area

The trick with negative area does not work for MSER output (!). And by the way it is also the function cv2.contourArea(cnt, oriented=True) which fails on Linux:

OpenCV Error: Unsupported format or combination of formats (The matrix can not
be converted to point sequence because of inappropriate element type) in
cvPointSeqFromMat, file
/build/buildd/opencv-2.4.2+dfsg/modules/imgproc/src/utils.cpp, line 59
0
Traceback (most recent call last):
  File "airrace.py", line 180, in 
    testPaVEVideo( filename )
  File "airrace.py", line 155, in testPaVEVideo
    print frameNumber( header )/15,  filterRectangles(processFrame( frame, debug=True ), minWidth=150/2)
  File "airrace.py", line 31, in processFrame
    area = cv2.contourArea(cnt, oriented=True)
cv2.error: /build/buildd/opencv-2.4.2+dfsg/modules/imgproc/src/utils.cpp:59:
error: (-210) The matrix can not be converted to point sequence because of
inappropriate element type in function cvPointSeqFromMat

Why it does not work is probably due to missing external contour, but it sometimes works, so I am not sure. If you want to play, here is the original picture, uncomment g_mser creation (airrace.py:27) and you can see yourself:

./airrace.py mser-orig.jpg

No, you won't see it! That's even more scary!! Forgot about oriented parameter for MSER area!!! The stored JPG file is a little bit different, but you get completely different results than from original video (see the sign for 1412):

m:\hg\robotika.cz\htdocs\competitions\robotchallenge\2014>m:\git\heidi\airrace.py mser-orig.jpg
1412.5
-3237.5
-7165.5
-897.5
-1488.0
-1261.0
492.5
-6224.0

vs.

m:\git\heidi>airrace.py logs\video_rec_140318_195005.bin 62
62 -6079.5
-3189.0
-1648.5
-32.5
-1202.0
-4846.5
-369.5
[((406, 49), (124, 23), -61), ((346, 209), (132, 23), -77)]

… I do not feel comfortable now :-(. For MSER just forget about area. It is filtered out at the beginning anyway, right?

g_mser = cv2.MSER( _delta = 10, _min_area=100, _max_area=300*50*2 )

I should probably learn more about the parameters…

March 23rd, 2014 - mosaic.py (5days remaining)

Couple days ago I wanted to play a little, so instead of reviewing more and more tests results, I tried to write mosaic.py — script for collection multiple images into one. The first plan was to feed the script with a list of images only, and the rest will be automatic. Very naive with my energy and mental level that evening. So I drop it that night.

Later I started to play with MCL (Monte-Carlo Localization) and one crucial "detail" is how big is the drone position estimation error? My only reference is list of the collected images, so mosaic.py implementation would have place even in this tight schedule . This time it was fully manual: press arrow keys for shifting new image and q/a keys for image rotation. It worked. But it was a tedious work even for three samples. And it did not quite solve the original task — how big is the error?

So I slightly changed the interface and instead of list of images I am passing there CSV file containing filename, x, y, degAngle on every line. It also prints this on standard output when you do the correction, so you can use it next time.

Well, so far so good. The struggle came with exporting CSV from airrace_drone.py. Y-coordinate on images is upside-down, rotation is not in the center but in top left corner, camera is rotated -90 degrees to robot/drone pose … and probably much more, like image size dependent on height or tilt compensation.

The result is still not worth presentation — it is mainly for me, to write the notes down, so I will realize where is the mistake?

not very nice mosaic

The images are added via cv2.add(img1,img2), so overlapping parts are brighter. The shift in the turn is probably due to scaling but at the moment I have no idea why the drone basically stops on the straight segment (white area at the bottom). Maybe I should manually correct the mosaic, so you get an idea how nice/useful it could be. BTW the absolute error in direction is very small (less then degree), so compass is great (if it works at all! — we should get replacement for Isabelle next week, ha ha ha).

So here is mosaic with shift correction only

mosaic with shift correction

Every image is single video I-frame, i.e. 1Hz and approximately 15seconds of flight. The black dash-line segments are 30cm long with 10cm spacing.

And here is the mosaic corrected with angle (13 degrees correction for the last image, i.e. 1 degree per frame):

mosaic with corrected angle

Exercise for you — how fast was Isabelle flying?

March 24th, 2014 - MSER and battery (4days remaining)

Here are some results of today testing:

MSER is superior to normal thresholding
there is again problem like before with normal contours and negative area:

Inverse MSER region
we finally completed 10 loops (see video), but Heidi battery died after 6 minutes

March 25th, 2014 - Bermuda Triangle (3days remaining)

My Bermuda Triangle is the center area, where the lines are crossing, and drones are loosing their navigation skills. The wrongly detected strip has following properties:

it is white
it is on the image border
it is triangle

At first I want to attack the 3rd point, triangle. The idea was: OK, if it is rectangle than minimum fitting rectangle will have almost the same area while triangle will have only half of it. Does it sound reasonable?

I verified 6 minutes video that I still get identical results as in the real flight yesterday (sometimes these "stupid" tests are the ones most important) and it was OK. So I added area computation and print:

rect = cv2.minAreaRect(cnt)
area = cv2.contourArea(cnt)
print "AREA %.2f\t%d\t%d" % (area/float(rect[1][0]*rect[1][1]), area, rect[1][0]*rect[1][1])

Well, I expected percentage of covered area, but …

AREA 0.05       121     2482
AREA 1.60       4936    3084
AREA 1.26       5033    3983
AREA 0.35       253     731
AREA 1.93       1830    950
AREA 0.57       1366    2415
AREA 2.24       6819    3044

… nice, isn't it . So what the hell is that?!

It looks like it is just an array of points, unsorted. Now it makes sense to apply convex hull as in one of MSER examples, before the contour is drawn. So I am not looking for area, but all I need is len(cnt)

AREA 0.90       2242    2482
AREA 0.91       2806    3084
AREA 0.88       3514    3983
AREA 0.86       626     731
AREA 0.83       787     950
AREA 0.90       2178    2415
AREA 0.93       2845    3044

That looks much better .

Do you want to see numbers for yesterday image (video_rec_140324_195834.bin, frame 17)?

AREA 0.58       1438    2475
AREA 0.57       1843    3219
AREA 0.26       4041    15301
AREA 0.32       5284    16650
AREA 1.01       624     620
AREA 0.99       818     825
AREA 0.86       1586    1850
AREA 0.85       2104    2481
AREA 0.87       1320    1519
AREA 0.85       1673    1964
AREA 0.82       2269    2772
AREA 0.91       1602    1767
AREA 0.84       2361    2794
AREA 0.91       3016    3299

First two are probably detected triangles for different threshold levels. After that is a big crossing area and then reasonable strips. So 70% would be maybe enough. And here is the result:

Filtering 70% rectangle area

And now it is time for repeated video processing with identity assert — will it fail? Yes, even the first very picture :-(. I had to add condition to skip first 10 frames (seconds) … during take-off there are many random objects. Then it went fine to frame 218 … hmm, but I do not see anything interesting there (*). Time to re-log whole video and compare text results.

OK, there are 17 different results, frames 5, 920, 1055, 1220, 1835, 2180, 2480, 2810, 3470, 3800, 4460, 5405, 5420, 6035, 6185, 6350, 6530 (divide it by 15 to get I-frame order). Boring? Who told you that this is going to be fun? Off topic — I can pass I-frame index to see/analyze only particular frame, but what should I do with these "times 15" numbers? Bad mathematician and lazy programmer end up with the following code:

testPaVEVideo( filename, onlyFrameNumber=int(eval(sys.argv[2])) )

The result looks OK, actually even better. It filters not only the bad triangles but also slightly inflated rectangles … so far so good. We will see during tonight tests. Here is the code diff.

(*) 218 was not frame number. It was number of updates between frame 905 and 920, so the first failing frame was in reality 920/15

March 26th, 2014 - Isabelle II (2days remaining)

Believe it or not, we have new drone now. It came yesterday as "hot swap" for Isabelle with broken compass. And she already flew several rounds … thanks Apple Store

The second most important progress is, that MSER now works also on Jakub's computer with Linux and OpenCV 2.4.2. The workaround is here (I am sure there is some faster/proper way to do it directly in NumPy, but there is not much time left for study/experiments).

The first test went as always: Heidi took-off, detected strip, started to turn, missed transition to straight segment and continued in the circle … damn. The second test went fine. This scenario is already repeated like 5(?) times … and the magic is … height. We added extra time to reach the 1.5m height and since that (knock, knock, knock) it works fine. Here is the next changeset.

There is left/right tilt compensation correction implemented now. Why? We had added strips filtering that they not only have to be spaced by 40cm and similar angle, but that Y-coordinate offset is smaller then 25cm. But then we have seen drone following line and on one image it is tilted to the right and on the second image (a second later) it is tilted to the left. The difference was only 6 degrees, but for 1.5meter height this means 15cm and that already matters.

Another "break through" (ha ha ha) came to mine mind on the bus on the way to CZU. We should not follow the line exactly but with an offset! That way we increase the chance to see transition line to circle. Here is the code.

OK, if you finished reading up to here … here is video from testing (not edited). You can see there whole team except me + one visitor.

Internal notes …

Now is time to review yesterday log files. There are 7 of them + 500MB from Jakub. First two logs are without offset, then for speed 0.4m/s, 0.6m/s with increasing step 0.05 per loop, 0.75m/s and 0.7m/s.

I am not able to replay meta_140325_192727.log … my notes are not very good … I see that was the moment when we had speed 0.4m/s, step 0.05 BUT there was constant angle correction independent on desiredSpeed. OK, now replay works without assert.

The last log ends nicely:

BATTERY LOW! 8
!DANGER! - video delay 1.097775

… and in 5 seconds it hit the column.

Now, refactoring … I promised to delete all unused code, sigh. It must be done. Now. MCL — gone. classifyPath — gone. PATH_UNKNOWN and PATH_CROSSING — gone. Diff looks now a bit scary. Hopefully three reference tests were enough for this refactoring. There is one more round of test tonight, so this was the last chance.

March 27th, 2014 - Sign mutation (1day remaining)

Are you also sometimes lazy to think about slightly more complicated mathematical formula that you just try it instead? And if it does not work, you randomly change signs? I would call it genetics programming (to complete the evolution you can add to this process also copy and paste). And sometimes it is not good idea…

I suppose that I had to use it for navigation along the right turn curve. It is like left turn, just the signs are opposite, right? Some mutants are quite healthy, and this one was for a relatively long time! Have a look at the video again and closely watch the right turns (1:01-1:16, 1:41-1:57). It is easier to hear it than to see it. What is the problem? We expected some magnetic anomalies, but the truth is that the signs were not quite optimal. I am not sure if I should be ashamed or happy that we found the bug . Simply for right/left angle correction was used the wrong side of the circle (imagine circle and its sides, … I mean there is a place where your error is close to zero and there is also the opposite place where you are close to math.pi, -math.pi singularity. It actually works (as you can see it on the video), except that you steer with maximum left, right, left, right …

OK, that was good one. There are two more issues, which I would call nightmares. I know about them, but I am not quite sure what to do about them:

losing WiFi connection
video delay

The first one looks like this:

Traceback (most recent call last):
  File "M:\git\heidi\airrace_drone.py", line 237, in 
    launcher.launch( sys.argv, AirRaceDrone, competeAirRace )
  File "M:\git\heidi\launcher.py", line 82, in launch
    task( robotFactory( replayLog=replayLog, metaLog=metaLog, console=console ))
  File "M:\git\heidi\airrace_drone.py", line 215, in competeAirRace
    drone.moveXYZA( sx, sy, sz, sa )
  File "M:\git\heidi\ardrone2.py", line 633, in moveXYZA
    self.movePCMD( -vy, -vx, vz, -va )
  File "M:\git\heidi\ardrone2.py", line 539, in movePCMD
    self.update("AT*PCMD=%i,1,"+ "%d,%d,%d,%d"%(f2i(leftRight),f2i(frontBack),f2
i(upDown),f2i(turn)) + "\r")
  File "M:\git\heidi\airrace_drone.py", line 79, in update
    ARDrone2.update( self, cmd )
  File "M:\git\heidi\ardrone2.py", line 258, in update
    Pdata = self.io.update( cmd % self.lastSeq )
rocess Process-1:
Traceback (most recent call last):
  File "C:\Python27\lib\multiprocessing\process.py", line 258, in _bootstrap
  File "M:\git\heidi\ardrone2.py", line 113, in update
Process Process-2:
Traceback (most recent call last):
  File "C:\Python27\lib\multiprocessing\process.py", line 258, in _bootstrap
    self.command.sendto(cmd, (HOST, COMMAND_PORT))
socket.error: [Errno 10065] A socket operation was attempted to an unreachable
host
    self.run()
  File "C:\Python27\lib\multiprocessing\process.py", line 114, in run
    self.run()
  File "C:\Python27\lib\multiprocessing\process.py", line 114, in run
     self._target(*self._args, * *self._kwargs)
   self._target(*self._args, * *self._kwargs)
  File "M:\git\heidi\video.py", line 30, in logVideoStream
     data = s.recv(10240)
e File "M:\git\heidi\video.py", line 30, in logVideoStream
rror: [Errno 10054] An existing connection was forcibly closed by the remote host
    data = s.recv(10240)
error: [Errno 10054] An existing connection was forcibly closed by the remote host

I wanted to somehow clean the dump for you, and I did not realize that it is actually two crashing processes with interlaced error output (*). One process is handling navdata and sending flight commands while the other is video processing. Both crashed on socket operation, where An existing connection was forcibly closed by the remote host.

This happens after several minutes flight from my Win7 notebook. The drone stops and is hovering at last position. Network is temporarily unavailable (in the worst case the notebook can switch to another WiFi access point, if you are not cautious enough). So what? Take a break and restart the whole program in 5 seconds? What about details like take-off, trim gyros or that first segment is not PATH_TURN_LEFT any more?

The second nightmare looks like this

NAVI-ON
!DANGER! - video delay 158.220822
FRAME 20 [-1.6 1.2] TRANS L -> I
I True 5
!DANGER! - video delay 158.392327
FRAME 22 [-1.6 1.2] I True 4
!DANGER! - video delay 158.393941
FRAME 24 [-1.6 1.2] I True 3
!DANGER! - video delay 158.490557
FRAME 26 [-1.6 1.2] SKIPPED2
I True 2
!DANGER! - video delay 158.642409

And here is the story how it happened: I asked Jakub to read configuration files form Heidi and Isabelle II, because for some unknown reasons Heidi is still faster. Maybe more experienced? My old hacked program crashed after the read, and it almost did not matter … except the recorded video was not transfered, so the drone tried to do that in the next run.

You can see similar reports from my tests last year. I.e. it is expected, but we should be careful about it. It is quite hard to navigate from more than 2 minutes old video .

(*) there are actually three failing processes: navdata, video and recorded video

March 28th, 2014 - WiFi Sentinel (0days remaining)

I cannot sleep again (the clock says 2:47am now) :-(. Yes, the nightmare(s) again. Yesterday I took „drone day off”, but Jakub did some more tests with WiFi failure. We had several iterations with wifisentinel.py, which is a script for securing WiFi connection. You write program arguments as usual except you put this wrapper at first place, for example:

python wifisentinel.py python airrace_drone.py test

The sentinel first verifies if you are connected via one of three allowed IP addresses. If yes then it starts the drone code otherwise it waits 3 seconds. When the drone code crashes, for example due to Unreachable host problem, it again waits until connection is reestablished and runs the code again and again, and again …

Probably the funniest bug was this one, when instead of drone flying code it was calling self again. Then there were some issues with shell=True … you can always put the shell name in front, so now it is shell=False when using the subprocess.call(). Finally the code for Python ipconfig is extremely ugly and fits only my and Jakub's operating system — I have no idea why he choose Czech Linux , where you get strange console outputs like:

…
inet adr:10.110.13.102  Všesměr:10.110.15.255 Maska:255.255.248.0
…
AKTIVOVÁNO VŠESMĚROVÉ_VYSÍLÁNÍ BĚŽÍ MULTICAST  MTU:1500 Metrika:1
…

If you know how to do this better, both on Linux and Windows without need of installing extra package, let me know. I will change it (not today and not tomorrow).

The test was following:

start drone program
turn WiFi off during flight
turn WiFi on again (after couple seconds)

The old code would crash immediately on the step 2 and drone would be hovering in place. Surprisingly you get the same bad result also with previous revision of the new code . Yeah, testing is important. In that case the navdata communication channel crashed, but both video processes remaining active! At least I learned that there is daemon value for multiprocess too. And then it went fine. Jakub repeated several WiFi failures and Isabelle always recovered.

So what is the problem? There is still video delay. Say, the drone recovers within 15 seconds. During that period it is recording area below it. It is slightly moving, and it is still recording. Then it starts to fly and the log file looks like:

NAVI-ON
USING VIRTUAL LEFT TURN CIRCLE!
!DANGER! - video delay 20.909436
FRAME 71 [-1.7 0.2] TRANS L -> I
I True 4
!DANGER! - video delay 20.406242
FRAME 72 [-1.7 0.2] I True 3
!DANGER! - video delay 19.874204
FRAME 73 [-1.7 0.2] I True 2
!DANGER! - video delay 19.346308
FRAME 74 [-1.7 0.2] I True 1
!DANGER! - video delay 18.818665
FRAME 75 [-1.7 0.2] I True 0
!DANGER! - video delay 18.260648

i.e. the delay is getting smaller and smaller as the drone is downloading the old video from period without connection. But it also tries to fly based on these video frames!!! Believe it or not, it could be OK for couple seconds until you have to switch from line to circle navigation. It is using only the old video as reference but it has no knowledge how far it is on the 8-loop.

So now the dilemma: change otherwise tested code or not? Was is just luck that it worked fine yesterday or the drone should rather stop and wait another couple seconds? It seems that it is capable to download 2s of video within 1s, so to decrease the video delay by second on every image … no, stupid me. There are different times. The time difference cannot be more than 1 second. If the processing would take 0.001s the images are recorded at fixed times, every second, so it would decrease it by 0.999s, but it would took 0.001s. So maybe it is not that bad.

Let's review this example FRAME 71: 20.9s delay and FRAME 103: 1.6s delay. So 32 frames and 19.3 seconds, 0.6s per frame, so approximately 0.4s it needs for processing on Jakub's older notebook. It also means that it took 0.4*32=12.8s in real time, at slow speed (0.4m/s) this means more than 5m! OK, I will change the code NOW!!

I am looking at the logs in detail and it is scary, see this picture:

View after reset during flight

The default configuration is to look forward, and one of the first steps is to switch to down pointing camera. But it will navigate based on detected strips for a while … I am not going to change this part anyway. So you can tell even from images that the drone was restarted.

Code for The Last Prague Test is ready and I can go sleep again … it is 4:46am, so I can at least try .

p.s. yesterday came very important info from Karim (Robot Challenge organization). The schedule is updated and Air Race --- Each robot has to pass the homologation in order to be allowed to participate in the competition. Please don't wait till the last minute during the homologation time. In the competition, each robot has multiple attempts - the best one counts. Please don't wait till the last minute with your runs. I am waiting for confirmation but in this case we could test STABLE (current) version and RISKY (to be prepared during drive to Vienna) version . And we have two drones to crash! Now I have reason why to take spare indoor protective hull . Držte nám palce

March 29th, 2014 - Falling Down (11 rounds)

It is over now. Do you know the movie Falling Down? Somehow it is the first association I have now . But let's start from the beginning …

… I just realized why the robot died, as always. Originally a good idea killed that …

Yesterday evening we have studied the last Friday tests, including speed profiles. Here is the graph for max desired speed set to 0.8m/s, dropping to 0.5m/s when no strip was detected and slowing down to 0.4m/s for transitions between turns and straight lines:

Desired and current speed

The robot at the end failed due to large shadows wrongly detected as strip. So we added also maxWidth limit. That was all we changed … we were too afraid and tired to do any other change.

This morning we did some more changes. First of all we set the radius to 1.25m, because it was smaller in Vienna than in Prague. We did first test and there were problems with video delay, crossing, and Isabelle height. So we decided to increase desiredHeight to 1.7m, limited max allowed video delay to 2s and implemented crossing detection. The crossing on Robot Challenge 2014 looked like this:

Crossing in Vienna

And that was it. That was the final code.

How it went I will leave for tomorrow morning … the hint, why robot crashed after completed 11 loops is already mentioned … but maybe not as Jakub just showed me the latest images.

p.s. Air Race 2014 - Isabelle CULS video is finally uploaded … again it is without censure and editors cuts, so sorry if you will hear some rude words …

March 31st, 2014 - 3rd place

Results before Isabelle start

Judge attact - end of Isabelle attempt

3rd place

April 1st, 2014 - Altitude, Pressure and Temperature

Yesterday I tried to investigate „the falling down” problem, which caused problems with navigation and Isabelle crash after 11th loop was completed. Here are some data for NAVDATA_ALTITUDE_TAG:

slowly falling down

It is a little bit hard to see but the sonar (green) and vision (red) graphs are almost identical. These are raw values in millimeters, if I remember correctly. The deadly reference has yellow color. Finally you can see also blue graph of maximal rectangle size (0 for no rectangle detected) and you can see that it was slowly growing but did not reach limit 200. So Jakub was right and we did not broke it with maxWidth filter set to 200.

So why is the yellow reference so wrong? Well, first of all to use altitude (drone.coord[2]) was not good idea. It is integrated over time also from PWM and pressure sensor. It has to compensate for terrain changes and detected obstacles (I know that the floor was flat, but Isabelle maybe did not know it). There was also small breeze in the hall, which could cause the first bigger drop (??).

The temperature was slowly increasing from 291 to 424 — no idea about resolution and units, so I would guess that it is 1/10th of degree. It was maybe getting hot due to heavier battery??

Anyway — temperature and pressure data are parsed now so you can use it in ARDrone2 class if you want.

I just replay some older test (meta_140326_201259.log) and the temperature there was 515, dropping to 510 and then rising to 525 … so I am not sure if it means 52 degrees of Celsius?! Could it be that if the drone is cold the estimation is not very reliable and we usually tested several times so the drone warmed up?? I am looking now at test meta_140325_180914.log and there is the reference estimate lower than readings from sonar and vision … so I would conclude, for Air Race do not use altitude, but only sonar and/or vision distance from the ground estimate.

Start view

p.s. if there will be next time we will have to look forward — the video footage would be much more interesting than to look down

April 11th, 2014 - Conclusion

Originally, I wanted to write more info about logs, wifi failures, competitors and plans for Air Race 2015. There is no time left and priorities sifted again … so I would like to thank the CZU team (mainly Jakub who did many many tests for several weeks — 9GB of log files).

Did we beat the last year best semi-autonomous score? No. We did not. But the winning team Sirin, with navigation strategy only above the cross, could have done that. They had times around 20s per loop, so 30 loops per 10 minutes. But they had also some problems … as everybody.

And what about Air Race 2015? Two Russian teams mentioned that they plan to build/use new drone, in particular Alex was mentioning PX4 platform. There is also a plan to modify the rules — we will see. I am definitely voting for multiple 10 minutes attempts . That way we will have chance to test stable and risky version and speed up if other competitors will also speed up.

p.s. if you would like to compete with your drone in a maze, here is another contest organized by Russian company … but there is age limit set to 18 … never mind … but it is the lower limit

contact form

Field Robot 2013

2012-10-23T00:00:00Z

The contest Field Robot Event 2013 will take place in Prague/Czech Republic for the first time. The main organizer will be Czech University of Life Sciences Prague (ČZU). The contest dates are 27th-29th June 2013, and the place will be very close to university grounds, i.e. Kamycka 129, Prague 6 - Suchdol. Registration extended until May 15th!

Robotour 2013

2012-11-08T00:00:00Z

Rules in PDF format: English, Czech.

Patronage

Robotem rovně 2010

2011-03-09T00:00:00Z

Cogito MART, robot Ambra

Rules

The goal of the competition: “To let the robot drive through the track without any interaction with his owner or anyone else and to stay on the pathway at all times. The road is about 3 meters wide and 300 meters long.” That’s just an excerpt from the rules, which you can find at Radioklub Pisek official website.

The robots competed in three rounds and in a given order, which has been determined by their velocity, measured at the approval (a test if the robot can pass at least a meter long track). The faster robots went first and the slower later, so they wouldn’t have to redundantly pass each other. The time difference between the starts was two minutes, so they went relatively quickly one after another.

A point was given for every meter the robot successfully drove, and the total from all the rounds was the final score.

First impressions

When I saw Radioklub Pisek had this competition last summer, I thought of it as a Robotour’s rival. But I see it way differently now – “Robotem rovně” (RR) could be an ideal preparation race for Robotour. RR gives a chance to complete beginners and mainly, lots of experience that you can use at Robotour. It was not a coincidence, that many teams from Robotour came to this event, to dust off their robots and train.

The event was very well organized and you could see that this wasn’t the first one, that Radioklub Pisek held. Even the relatively fast tempo of the competition was acceptable – observers weren’t bored, something was going on all the time, and the competitors also had a good time those three rounds.

Competition difficulty

The result of odometry

Even though it doesn’t seem at first sight, the competition difficulty is pretty high. To be more precise, the further you want to go, the harder it is. Almost all the cars and robots make it 10 meters. The width to length ratio of 3m to 10m gives you about 17 degrees range, so it’s no real problem to point your vehicle that way.

First round from the GPS’s point of view

If your ambitions are a little bit higher, you need a bit of luck, or some sensors. If you were thinking about GPS, you can give up already. There are some beautiful, huge trees in the park and even though you have signal from up to 9 satellites, the error isn’t going to be below 5 meters. I think that a demonstration sample from NMEA at Google maps (see pic.) will give you an idea. Necessary to say, the worst GPS position was on the start (then at least led the same way as the road), but you can forget about telling on which side of the road you are. After the import to OpenStreetMap, you can clearly see the uncertainty level (circles).

A GPS error in JOSM

Probably the next sensor, that comes to your mind, is a compass. That will propel you way further, but some surprise will show up also. Firstly, you’ll most likely have just a 2D compass, because they’re cheaper and more available. Which wouldn’t matter much, but you have to know what tilting does to the sensor. The road starts sloping downwards and, even if your compass can handle this, is like any other road slightly tilted to one side because of the rain.

Aside the tilt of the compass, which can be compensated, pay attention to the cables and the sewers hidden underneath the road, they will turn your compass into a roulette. So, you can use one, but it would be best to combine with odometry and gyroscopes.

A look from the camera at the start

The only remaining thing from commonly used sensors, is camera. The race consisted of three rounds, and each one of them was unique. The weather at 10 a.m. was cloudy, at 11 a.m. rainy and at 12.a.m. the sun was shining sharp. How’s Radioklub Pisek doing that, so you could train in all possible weather conditions, I don’t know .

About 100 meters from the start, there’s a past for camera too. You will arrive on a small, open atrium, and now search, kiddo. There was a little unclarity in the rules, whether the robot can use the side ways: path is path, but I guess the organizers meant to stay on the main path. Well, it isn’t simple.

But a proof, that you can reach the end, brought us Roboauto, that got the maximum possible score. The road was 300m last year, but this time it was cut down to 200m because of excavation works.

Results

The table of results is taken from the article robotem-rovne-2010-uspesne-za-nami

Results

R1 — Quido

Tomáš Ondráček/Roboauto

Roboauto

We took the competition in Písek as an opportunity to test our new platform: robot Quido based on an RC model wheelframe. It used to be a second car following a target on the lead “Karlík”, but we have upgraded it to work autonomously.

From cooperation with FIT VUT in Brno, our team received lidar SICK LMS 100 for the Sick Robot Day 2010 competition. We attached it to Quido not statically, but to a turning hinge, so that it could turn during the ride, and cover most of the area this way. Other than a camera we also used an odometer and a compass. We turned off the GPS for the race, because it wasn’t necessary and it would probably just complicate everything. For the same reasons, the front sonar was turned off.

The driving software was the same we used at Robotour. Lidar and camera took care of way detection, and the information was aggregated into a 2D map, in which the robot planned his way using a reactivating algorithm. No global map was necessary, considering the spirit of the competition.

To make the robot not to turn any undesired way on crossroads and large paved areas, we added an option to set a preferred stable direction in the way planner (which otherwise changes dynamically along the position in the RNDF map).

Last Friday night before the competition we arrived to the park in Písek with an intention to try the road once and to “smooth tune constants” from the recording whole night long. To our surprise Quido went forth and back the whole way without any doubts, so we could go to check the city square and some local restaurants :). On Saturday the same thing repeated, Quido had no malfunctions, so we ended up getting the maximum number of points.

The competition is, in my opinion, a good preparation for this year’s Robotour in autumn and can be a nice first public test for beginning roboticians. Radioklub Pisek did a great job in organizing this year, took care of nice weather, and a big thanks belongs to them for that.

R3 — Irena

Kamil Řezáč/Sirael

It might sound defeatistly, but I didn’t come to Písek to win. I took the competition primarily as something that will make me finish the robot in time. Which I had, with a couple of reductions, been able to do, but I didn’t have time to put together some advanced sensors or algorithms. So I decided to create a very simple algorithm – to control the driving unit (turning of the front wheels) according to the compass – basically a P regulator. To avoid problems with the absolute value of azimuth, the error caused by compass tilting and so on, I sampled and averaged the data from after the start from the compass (50 samples) and I drove by that value.

Originally I planned to not change the algorithm (which was tested one evening on the sidewalk in front of my house) at all. That didn’t go very well, the regulation loop was oscillating more and more. Despite all that, I started the first round with the original program, a poor result (25 m) was corresponding. So I reduced the “strengthening” of the regulation loop, the robot calmed down and the distance passed almost doubled (42 m). Because I saw the robot trying to turn right a little, I set it it turned slightly left the last round and the result was another increase in score (80m). The robot could go a couple meters beyond that, but his destiny has been cut off by a collision with the 80m sign…

I think this is the best (or close to it), that a “blind” robot can do, and it surpassed my expectations by far (best test try result was 42m).

R4 — Eduro Maxi

Eduro Maxi

Martin Dlouhý/Eduro Team

This was the first of the outdoor competitions this year for “Maxík”. It’s still a baby, which had driven his first meter on Monday in a kitchen, then a couple more meters after closing the bottom on Wednesday, and finally on Saturday, it went on his first race.

Robot was homologed using version 0 – go straight using the odometry. Version 1 then combined compass and version 2 added data from a camera and GPS. In the end, the robot was using a customized version 2, he drove straight for 1m, used a combined data from camera and the compass, turned the right way, and went straight for another meter. No rush, but you could easily see when the sensors started to feel, that the robot is going off the road.

The “Robotem rovně” was a great competition. It provoked us to make the robot work, so it will hopefully have a chance to score some points at Field Robot Event. A week after that is RoboOrienteering, and well, Robotour 2010 in Bratislava in autumn and SICK Robot Day in Germany. Once again, huge thanks belong to the organizers for a great event.

R7 — ARbot

Aleš Ruda/ARbot

The robot went through a big changeover after last year’s Robotour. New, more accurate GPS, AHRS instead of a compass, new optics on the camera, writing parts of the SW into assembly, a new barrel holder and a lot of little things. Unfortunately, the MD23 motor units started to stiffen. Subsequently I found out, that the cause were high voltage peaks in the 5V supply branch. Two months of suffering came to an end in the beginning of May, when I upgraded to MD25. Last day before the competition, the robot started moving again, some errors in SW had been fixed and he even drove through a curvy way towards my house, so “Let’s go!” to Písek in the morning.

A view from the robot

Little of time, no space to be a hero. The strategy was, that a simple camera will keep the robot on the track and sonar will assist with obstacles. It worked for Robotour, anyways.

Quickly test the robot in local ambience and to the start of the first round. 3, 2, 1, start. The robot started moving, but instead of going straight on the way, he “drunkishly” went from side to side, but he made it to the end. Greeeeat. What’s with the weaving, though? Oh, I forgot the camera lens cover.

I tried to make it go with the camera between rounds, but the results weren’t good. Robot went off the road very soon. I couldn’t fix the road detection algorithm. I tried to make the AHRS work, but didn’t face much success. So, if it worked once, why not twice more? Robot will go according to the sonar. Next round 90 and 184 meters. Good job. I wouldn’t believe this if someone told it to me, but it’s true. I saw it with my own two eyes.

I think that this competition was a great start of the outdoor robot contests. Thanks to organizers for the attitude.

R9 — Brimstone

Žlutásek

Michal Eliáš and Monika Svědirohová

The “Robotem rovně” competition presented to us the first experience with outdoor navigation. The wheelcase of the robot is Monika’s Žluťásek, I was left with the algorithms and track planning. I used a convolution principle from the webcamera, according to which our robot drove, no other sensors were used. The principle is to drive through the park manually, when the camera takes pictures at a given rate, while the picture isn’t saved as a whole, just as vertical columns of information calculated from the picture. Then on the competition the robots takes pictures at the same rate, and compares them to ones from before. This method is very memory and processor demanding, it needs at least an ARM7 level processor. We have been able to go 152 meters, but testing before, we always passed the whole track. The reason for not passing the track was a vulnerability of the algorithm to spectators standing alongside the track. We came fifth in the end, which is a great success, considering the fact, that the first test of the algorithm was made in Písek.

R11 — Ambra, R13 - UTrooper

Jiří Iša/Cogito MART

Our faculty (MFF UK) has bought a new outdoor robotic wheelcase this year – Utrooper. The goal (achieved) for Písek’s “Robotem rovně” was to test it in a competition and to find some of it’s attributes and flaws. Flaws were the bigger piece, but that can be expected with a new untested wheelcase.

The other robot taking part in the competition was Ambra – a customized RC Hummer, originally built for Robotou 2009. Ambra went through a big hardware change this year. So it’s basically a new robot’s premiere. Even this robot showed a nice flaw, after two pretty successful rounds. But altogether Ambra drove pretty well, plus got the vote of the 2nd vicemiss from the spectators!

We recommend the “Robotem rovně” competition to teams, that find Robotour too difficult, but also teams, that have bigger ambition, but have a new robot. Thanks to the organizers for a well held event in a beautiful neighbourhood

Final words

Robots

We recommend the “Robotem rovně a.k.a Autíčka v parku” competition warmly to everybody. Other than a good event it is a great place to meet robotics and roboticians, and full afternoon is free, so you can chat a bit. If you missed this event, the next outdoor autonomous robots contest is RoboOrienteering in Rychnov nad Kněžnou. Finally, if you’re from Písek or its surroundings and robots are one of your interests or hobbies, don’t be shy to contact Radioklub Písek, they’ll be glad to assist you, or take you into their team.

References:

Organizers website: http://www.kufr.cz/

Contact form

PocketBot

2010-05-06T00:00:00Z

PocketBot

dimensions: 48 × 32 × 12 mm
weight: 19 g (body 13g, cells 6g)
speed: 0.35 m/s (line following) 0.6 m/s (maximal)

The robot was primary designed to fit into a matchbox. A homemade double-sided printed circuit board stands as the robot’s chassis at the same time. Robot is powered with two rechargeable lithium-ion button batteries wired in parallel (3.6V, 40mAh each). The Atmel ATmega8 microcontroller runs robot’s program, which is written in C. An 8-pin connector offers ISP and UART interface for programming and debugging, respectively.

Undercarriage

Two separately driven wheels (8mm diameter) provide differential steering. The dimensions of the gear mechanism were crucial due to considerable space constraints. Fortunately, I met Josef Vandělík who designed and manufactured the wheelframe for my robot. [3] The wheelframe employs a friction gear system with magnetic pressure. A neodymium magnet in the central tube attracts wheel axles, pressing each wheel to the motor shaft.

Top View

Line following

The robot is capable of line following. That means it is able to follow a black guiding line marked on bright surface. There might be line crossing on the track, in such case the robot chooses a straight direction. The robot can avoid obstacles on the way; if an obstacle is detected, the robot reverses and continues backwards.

Finally, the robot can find a guiding line in unknown environment. When there is no line present under the robot, the robot starts to seek on a spiral trajectory until it crosses a guiding line.

Bottom View

Line sensor module

A guideline is marked with a black PVC isolation tape. This material doesn’t reflect infrared light so it is easy to distinguish a guideline by a light reflection of the surface.

The sensor module consists of 3 detectors and 4 emitters. The emitters (infra-red LEDs) and detectors (phototransistors) are placed in a row alternately so that each phototransistor is surrounded with two IR LEDs. Thanks to this design it is possible to measure the surface reflexivity on six spots under the sensor module, using only three phototransistors and four IR LEDs. Generally, this approach reduces the number of components and ADC inputs required for a line sensor module, which is desired with respect to dimension constraints.

The illustration bellow shows how this method works:

LED1 is emitting infrared light that reflects to the phototransistor T1, hence the light reflexivity at point 1 is measured.

Then, LED1 is turned off and LED2 starts emitting IR light. The phototransistor T1 measures the light reflexivity at point 2.

In real application (especially when high refresh rate is desired) the characteristics of IR components must be taken into account. Due to reaction delays (latency), data from sensor might be biased if the reflexivity at point 2 is measured right away after measuring the reflexivity at point 1. To keep from this, I analysed sensor characteristics on an oscilloscope and then I modified the scanning sequence in a way so that the sensors do not interact with each other.

Ambient light suppression

Because light conditions often vary according to time and place, it is necessary to use an ambient light suppression algorithm for sensors to work properly. The method is simple: Every sensor does two measurements. At first, it scans for the amount of ambient light. Then, it turns its infra-red LED on and measures the value again. Subtracting these two values, the bias of ambient light is suppressed.

Sensor calibration

There might be slight differences in characteristics of individual optical components; therefore the sensor module should be calibrated. The calibration is done manually in two steps:

1. Offset calibration

All sensors are placed above the black guiding line. Once the calibration command is received, all sensors measure the surface reflexivity and measured values are stored in memory. Those are the offset calibration values. From now on, all measured values are automatically corrected with this offset. (Each time a measurement is made, the offset is simply subtracted from the actual value). As a result, all sensors will return equal value when they are located above the black line.

2. Gain calibration

During the gain calibration all sensors are placed above a white surface. Some sensors might be more sensitive than others, so the measured values differ from each other. But because the surface reflexivity under the sensor module is supposed to be equal, the gain coefficients for each sensor can be easily calculated. For future measurements, every measured value will be corrected (multiplicated) with its gain coefficient; so that all calibrated sensors will have similar characteristics.

Consequently, the calibrated sensor module will output normalized values.

Processing data from sensors, motor control

Optical sensors measure the light reflexivity of the surface and acquired data is processed by the line detection algorithm. The algorithm is designed in such a manner that line width doesn’t matter. The line detection algorithm outputs signed integer value that states the actual deflection of a guideline. Values close to zero mean that the line is located accurately in the middle of the sensor module, positive values state how much does the line deflects to the right and negative values state the deflection to the left.

This output is then used for proportional-integral-derivate (PID) control of the line tracking. The PID controller adjusts the motors’ speed according to the actual line deflection and previous states. The position of the line is evaluated 30 times per second.

In other words, the PID controller drives the robot so that the line is always centered to the middle of the sensor module, so that the robot performs smooth line following.

Remote controlling

Remote Control

Robot is equipped with an infra-red remote control receiver. Therefore it can be controlled with a standard remote control or from a PC application. The wireless link to the robot is used for adjusting parameters (such as speed and PID constants), for sending sensor calibration commands and also for a manual operation. The wireless communication utilizes the NEC remote control protocol. [4] This protocol was implemented in both PocketBot (as a receiver) and USBdockStation device (as a transmitter).

Sensors

Remote Control

USBdockStation

The USB device ensures both wire and wireless communication between the computer and PocketBot. It’s based on an AVR-CDC project, a USB to UART converter. [2] Once the device is connected to a computer, operation system creates a virtual COM port which can be accessed from a computer application. PocketBot, USBdockStation, and the PC application communicate with each other through this UART interface, using a particular protocol that was designed for this purpose.

Top View

To control PocketBot remotely, I added some extra functionality to the original AVR-CDC firmware: The USBdockStation can emulate an infra-red remote control. It has a power infra-red LED and it is capable of sending IR NEC remote control packets as an ordinary remote control. It is obvious that the wireless communication is only unidirectional.

Bottom View

Firmware is programmed in C and it runs on ATmega8 with external 16MHz oscillator. The printed circuit board has custom design to fulfill needs of my project and it was manufactured industrially.

Communication diagram

PocketBot Control Panel

The PC application offers sensor diagnostic; it shows real-time visualization of sensor module state, including the assumed line position. Moreover, the application has also capabilities of sending wireless commands to the robot, which gives the key feature of adjusting PID controller constants remotely. Thus, PID constants can be tuned on-line during line following. On the other hand, wireless communication can be used for manual operation as well.

After startup, the application automatically scans over all COM ports for USBdockStation presence. Then it can detect whether the robot is connected with a cable, or whether only wireless communication mode is available. In bi-directional (cable) mode the battery fitness and calibration data can be accessed.

The application is programmed in Borland Delphi and runs on Windows operation systems.

Project timeline in brief

2007/06 I have started programming microcontrollers.
2007/10 I have designed and put up together my first line following robot [9]
2007/12 Project PocketBot started. I got inspired by the Desktop Line Following robot [1]. I built sensor module prototype and I designed the line position algorithm. Ambient light suppression method tested.
2008/01 Sony remote control protocol decoding, UART debugging, PWM motor control.
2008/03 Robot’s undercarriage finished
2008/04 Lithium-ion charger built. I won first place at the Student Scientific Competition with my first line following robot.
2008/05 PocketBot linefollower finished. Line following with proporcional (P) regulator [5]
2008/06 I presented paper titled Line following robots [11] at the international conference VIASL [10]
2008/07 PD regulator allows line following at higher speed [6]
2009/02 I implemented new NEC remote control protocol into the robot.
2009/04 USBdockStation finished. Sensor gain and offset calibration. Working on optimized switching sequence for sensors.
2009/05 PocketBot Control Panel finished. Graduation at high school in Prague.
2010/01 Remote control protocol analyzer finished [7]
2010/03 PocketBot won first place in the freestyle category on RobotChallenge competition in Vienna. [8]

In Box

Author

Ondřej Staněk (20) is the first year student of the Computer science at the Faculty of Mathematics and Physics at Charles University in Prague, the Czech Republic. He is interested in computer programming and electronics. Apart from mobile robotics, the author participates in a development of a snow measuring device for hydrometeorology science.

http://www.ostan.cz, ostan89-at-gmail.com

References

[1] ChaN’s Desktop line following robot: http://elm-chan.org/works/ltc/report.html

[2] Software USB to UART converter for AVR microcontrollers: http://www.recursion.jp/avrcdc/

[3] Author of pocketBot’s wheelframe: http://www.volny.cz/jova3/pocket_bot/pocket_bot.htm

[4] NEC protocol description: http://www.sbprojects.com/knowledge/ir/nec.htm

[5] PocketBot video I.: http://vimeo.com/6394938

[6] PocketBot video II.: http://www.youtube.com/watch?v=8UCQyGJ5M0E

[7] Remote control protocol decoding application: http://ostan.cz/IR_protocol_analyzer/

[8] http://www.robotchallenge.org/scripts/trunk/trainstation/detail.php?hide_controls=true&refresh=99999&competition=126

[9] My first line following robot: http://ostan.cz/robot/ipage00013.htm

[10] The VIASL conference: http://www.ifip2008praha.cz/

[11] Submission paper for the VIASL conference: http://www.ostan.cz/robot/line_following_robots.doc

Comments and questions form

Brimstone

2010-03-02T00:00:00Z

Cybernetic car ("Brimstone")

Construction

Robot

Robot chassis was based on the Lynxmotion kit. Its a 4x4 car with "tank" like motion.

Brimstone works in three modes :

Bluetooth remote control, where I command the robot and it detects obstacles. If an obstacle is detected, it stops and waits for further commands.
Fully automatic mode, where it an actively bypass obstacles.
Fetching mode. Robot is remote controlled and picks up objects. Obstacle detection does not stop the robot but slows it down so the robot can pick up the object easily.

Bluetooth remote protocol allows control of my robot by a PC, PDA or a cellphone. Bluetooth range, which is approximately 100m is sufficient for most environments.

Software

Remote control

The program is written for ATmega32 in C language. The code implements a "state machine", which works as follows :

robot receives a command via a serial line and performs action based on the command and its current state. The action can change current state of the robot.
example: current state - robot moves backwards, received command - go forward, resulting action - stop and then move forward (stopping before reversing speed preserves both motors and control boards

Autonomous mode

This mode uses interrupts generated by A/D converters attached to robot sensors. As soon as the sensors detect an obstacle, an interrupt is generated and obstacle avoidance is activated. Further interrupts are inhibited and robot starts to avoid obstacles. Obstacle avoidance routines are based on state machine too. Autonomous mode is terminated by an interrupt generated by a serial communication interface.

Construction:

soldering

robot from inside

wiring

robot from above

gripper

eyes in the dark

Eurobot

After improvement of the mechanical hand I tried to participate on contest EUROBOT 2009 in category STARTER, where I surprised the audience by moving the robot over the playing elements instead of picking them up. My robot was not able to build any "temple", but I gained valuable experience by participation.

Video

Created by: Monika Svědirohová

Monika

demo

If you have any questions or comments – contact us.

Robotika.cz

OSGAR

John Deere X300R

Team

2014

14th May, 2016 — Contest „Robot, go straight!”

4th June, 2016 — Contest „RoboOrienteering”

Video from RoboOrienteering 2016

23rd November, 2016 — Hydromotor video

16th May, 2017 — „Robotem rovně” (again) and the first garden test

Links

Robotour 2017

Rules

Examples of roads in Žilina (Ľudovít Štúr park)

Location

SICK Robot Day 2016

Rules in nutshell

Results

Cogito MART (1st place)

CS Freiburg (2nd place)

Gripper

Cube Perception

RFID Localization

Behavior

Challenges during the competition

EDURO Team (3rd place)

Eduro "show time"

Results

Competition

Attendance

The Place

The Weather

Homologation

Runs

Total Score

Conclusions

ARBot photos (2nd Run)

Kamaro Betelgeuse Video

Introduction of teams

Teams

YouTube playlist of all registrations 2016

AmBot (CZ)

ARBot (CZ)

Cogito (CH)

Istrobotics (SK)

JECC (DE)

Kamaro Beteigeuze (DE)

Lois, JECC (DE)

MART (CZ)

No! This is Patrick! (DE)

Quadrons (PL)

Radioklub Písek - TCVVI (CZ)

Raptors (PL)

Smely Zajko (SK)

WallE (DE)

Robotour 2016

Rules

Examples of roads in Deggendorf

Location

Tour the Stairs 2015

Rules for year 2015

UPDATES

1st November 2015 — Registration video KRA Písek

9th November 2015 — Husky won't make it

28th November 2015 — Results and Lamia videos

Lamia - Tour the Stairs 2015 (straight)

Lamia - Tour the Stairs 2015 (twisted)

7th January 2016 — minivideo

KRA Písek

Lamia

Czech TV

Results

Competition

Attendance

The Place

0th and 1st Run

2nd Run

3rd Run

4th Run

Total Score