Huskylens – AI Machine Vision Sensor

Machine vision is a fascinating subject that, until recently, required a lot of expensive computer “horsepower”, as well as an understanding of complex algorithms and coding.  But today’s new generation of powerful microcontrollers has changed all of that, and there are now several low-cost machine vision solutions that are perfect for hobbyists and experimenters.

I have a few of those solutions to share with you and the first one, which is presented here today, is a very cool product from DFRobot called the HuskyLens.

Introduction

We have already looked at a machine vision product, the PixyCam. This little board has an onboard camera and was capable of simple image recognition and line following, among other features.

The board we are going to examine today, the HuskyLens, is similar to the PixyCam, but much more powerful. It has both a camera and a small color display and can be used as a stand-alone product or as a peripheral for a microcontroller or microcomputer.

This device is ideal for a number of image recognition and tracking applications.

HuskyLens

HuskyLens was brought to life as a Kickstarter campaign in 2020. It is now for sale on the DFRobot website, as well as through many electronics distributors (I purchased mine at DigiKey).

The HuskyLens is available in two versions, the only difference between the two is the video sensor, which determines the resolution of the camera:

• HuskyLens – OV2640 (2 MP camera)
• HuskyLens PRO – OV5640 (5 MP camera)

The HuskyLens that I am using is the 2 MP camera model.

Features & Specifications

Both models of the HuskyLens have similar specifications. The devices can be used with either 3.3-volt or 5-volt power. At 3.3-volts they consume an average of 320mA, whereas operation under 5-volts consumes 230ma (both figures quoted in face recognition mode).

The display on the HuskyLens is a 2-inch IPS screen with a resolution of 320 x 240 pixels. The module itself measures 52mm x 44.5mm.

The HuskyLens can communicate with your microcontroller or microcomputer using either I2C or a serial UART connection.  The 4-pin JST connector on the board is used for both protocols, and the device can either be set for one or can auto-detect the protocol on its own.

DFRobot provides a JST cable with four female DuPont pins on the other end, to simplify connecting the HuskyLens to a microcontroller or microcomputer.

Power for the HuskyLens can be provided through either the two power pins on the 4-pin JST connector or with a microUSB connector. If both power sources are connected, the USB power will be used.

The HuskyLens also has a microSD card slot. The microSD card can be used to store images, it can also store

The HuskyLens comes with a silicone cover (with “Husky ears”) and a mounting bracket.

Kendryte K210

The heart of the HuskyLens is the powerful Kendryte K210 microcontroller.

This chip has a 64bit RISC-V Dual Core CPU. It features Kendryte’s self-developed “KPU”, a high-performance neural network hardware accelerator that can perform convolutional neural network operations. 

The chip also has a high-performance microphone array audio processor that can simultaneously process the inputs from up to eight microphones.

The K210 is highly suited for image recognition and speech recognition applications.

Controlling the HuskyLens

There are two controls on the HuskyLens that can be used in stand-alone mode. The device can also be controlled programmatically using the attached microcontroller or microcomputer. 

One of the controls is a pushbutton, the other is a “dial” that can also function as a pushbutton.

These buttons both have multiple uses, which will be described in detail further in this article.

HuskyLens Firmware Update

Before you start using your HuskyLens you should check the current firmware version. If it is less than 0.5.1Norm then you should update the firmware.

You’ll need to connect the HuskyLens to a computer to perform the firmware update. Although you can find instructions for Linux, Mac, and Windows on the DFRobot HuskyLens Wiki, you are going to find it much easier to use Windows for the firmware update.  That is because there is a GUI updater available for Windows, whereas the other operating systems require you to use the command line.

Assuming you are using windows, you’ll need to download two things from the HuskyLens GitHub page:

• The Windows Firmware Updater
• The firmware itself – this is a link to version 0.5.1Norm

You will also need to download a USB to UART bridge, which you can get from Silicon Labs.

The procedure for performing the firmware update under Microsoft Windows is as follows:

• Start with the HuskyLens disconnected from the computer.
• Download the required files from the links above and unzip them.
• Run the USB to UART bridge.
• Plug the HuskyLens into a USB port
• Check your Device Manager and confirm that you see a “Silicon Labs” device connected to a USB port.
• Install the HuskyLens Updater.
• Run the Updater. When prompted to choose a firmware file, select the one you have downloaded.
• The upgrade process will begin

The firmware update can take at least five minutes, the graph on the updater will go from 0 to 100 several times during the process. Just be patient, and your HuskyLens update will succeed.

Once the update is complete, check the HuskyLens firmware version again. It should show the updated version now.

Modes of Operation

The HuskyLens has several modes of operation:

• Face Recognition
• Object Tracking
• Object Recognition
• Line Tracking
• Color Recognition
• Tag Recognition
• Object Classification

Let’s examine each of these modes in more detail.

Face Recognition

In this mode, the HuskyLens can detect faces, assign an identifier to them, recognize them afterward, and track their movements.

The device is capable of learning faces individually or in groups.

Learning a Face

We’ll begin by placing the HuskyLens in Face Recognition mode, which is accomplished by turning the selector dial at the top until “Face Recognition” appears on the menu at the top of the screen.

The screen will have a “cross” cursor at the center. Point that at a face you want to detect. When the HuskyLens detects the shape of a face, it will place a box around it, with the word “Face” at the top.

Now briefly press the Learning button (the pushbutton on the top right) and the HuskyLens will learn the face and save it with a label of “Face:ID0” for the first face.  Subsequent faces will be labeled “Face:ID1”, “Face:ID2” etc.

Although the face has been saved, it will only be recognized when viewed at the same angle as the sample face.  In the “real world” this will lead to poor performance.

In order to better recognize a three-dimensional face or a moving face in a video, you can train it for a longer period of time.

Deleting a Learned Face

By default, you can only learn one face. Other faces will be recognized as “face”, but they can’t be learned.

We can set the HuskyLens into Multiple face mode to recognize more than one face. However, in order to do that, we will first need to delete the face we have already learned.

In order to delete a face, or anything you have learned, you do the following:

• Get the face you want to delete into the screen where it is recognized. It does not need to be in the center, as long as HuskyLens can identify it, then it can be anywhere on the screen.
• Click the Learning key. A menu will pop up, asking if you would like to forget the identified object, in this case the face.
• Respond Yes using the wheel control.  The face will no longer be recognized.

Learning a group of faces

Now that we have a clean slate, we can set up the HuskyLens to recognize multiple faces.

The process of setting up the HuskyLens for learning multiple objects is identical for all HuskyLens modes, not just Face Tracking:

• Enter the Face Tracking mode using the wheel control and observe the top of the display.
• Press down on the wheel – you should see a new display at the bottom of the screen.
• Now use this new screen to navigate to Learn Multiple.
• Press down on the wheel – you should see a toggle switch display on the screen.
• Use the wheel to toggle the switch, it is ON when set to the right and colored blue. OFF is when it is to the left and colored white.
• Set the switch ON for learning multiple items.
• Use the wheel to navigate back to Save & Return.
• Press down on the wheel on Save & Return. Confirm by pressing down again.

Now the HuskyLens is set up to track multiple faces.

Track the first face as you did for single faces, aim the crosshair mark at the target face, and press down on the Learning key. Hold it down a whale to capture the face from different angles. Then release it, and the first face will be assigned an ID number.

You will get a message asking if you want to learn another face. If you do, then click on the Learning key once to confirm.

If you confirm that you want to learn another face, you’ll get the crosshairs again, which you can aim at another face. Once again, press the Learning key to learn the face. 

Continue this process to memorize as many faces as you wish.

Once you are done learning faces, you can either press down upon the navigation wheel or just wait until the message time out.

The HuskyLens can now identify multiple faces.

Object Tracking

In Object Tracking mode, you can train the HuskyLens to learn one object. The object can be just about anything, and even gestures count as objects.

You can enter Object Tracking mode by navigating with the selector wheel until Object Tracking is displayed on the top panel. Press down upon the selector wheel to select Object Tracking.

In order to learn a new object, you need to place the HuskyLens into learn mode. Press down upon the selector wheel and another menu will appear at the bottom of the screen.

Navigate to Learn Enable, which is another switch control. Use the selector wheel to activate Learn Enable.

Perform the same operation with the Autosave option, which you also want to be activated.

Now navigate back to Save & Return and press the selector wheel twice to accept and confirm your changes.

Now you’re ready to start learning an object. Aim the HuskyLens at the target object and press down and hold the Learning key. Keep the key pressed as you scan the object from several angles.

After you release the Learning key, the HuskyLens will recognize the object and will be able to differentiate it from similar objects.

The HuskyLens will continue to learn the saved object every time that it sees it, for improved performance.

Object Recognition

The HuskyLens has been programmed to recognize twenty different objects, as follows:

1. Airplane
2. Bicycle
3. Bird
4. Boat
5. Bottle
6. Bus
7. Car
8. Cat
9. Chair
10. Cow
11. Dog
12. Horse
13. Motorbike
14. Person
15. Plant
16. Sheep
17. Sofa
18. Table
19. Train
20. Television

You just need to select Object Recognition using the selector wheel and point it at your target area. If one of the preprogrammed objects appears in the HuskyLens field of view, it will be displayed and tracked. Multiple objects of different types are supported.

Here are a few samples, for more please see the accompanying video.

As you can see from the above samples, the HuskyLens does a pretty admirable job of object recognition.

Line Tracking

In Line Tracking mode, the HuskyLens can be programmed to recognize and track one or more colored lines.

Although the default mode is set to tracking multiple lines, you’ll often find it easier to track single lines, so you should turn off Learn Multiple unless you really need it. You can do that in the same fashion as you did for the other modes, select the mode, press the selector switch and navigate through the lower menu to Learn Multiple, where you can change the setting with the toggle switch.

You might also want to activate the white LEDs while you are in the bottom menu, as they can assist in line tracking.  They use a toggle switch as well.

Once you have made your changes, you can navigate to Save & Return to confirm your settings and exit.

Now you can begin to learn lines. Focus the HuskyLens on the line you want to use and press the Learning key. Hold it down while you track the line from various angles.

As you hold down the Learning key, the HuskyLens will display a yellow arrow in the direction of the target line, this indicates that it is learning the line. Once you release the key, it will display the tracked line in blue.

You will notice that it is quite good at tracking vertical lines, but not so good at horizontal ones. For most line tracking applications, this is the preferred operation.

Color Recognition

In Color Recognition mode, the HuskyLens can learn and distinguish several colors.

Like most of the other modes, you can set up the HuskyLens to recognize one or many colors, and the default is just one. To set it up for multiple colors, dial up Learn Multiple on the bottom menu and use the toggle switch to select this.

Now aim the HuskyLens at the first color and pres the Learning key down. If the color is on a flat surface, then you just need to give the key a brief press.

After you release the key, you’ll be prompted to click again to learn another color – well technically you’re prompted to “Clink” again, but I think you know what they mean!  Continue to memorize as many colors as you need.

After you finish learning colors, press down on the selector dial to exit learning mode.

Now the HuskyLens will identify the memorized colors whenever it sees them.  Note that lighting conditions can play a big part in the accuracy of this prowess, so if you’re having trouble getting reliable results, try changing or moving the light source.

Tag Recognition

AprilTags are a creation of the University of Michigan April Robotics laboratory. They can be used to identify areas, objects, or actions and serve as calibration targets for robots.

The HuskyLens can learn and identify multiple AprilTags. It does not support any other type of tagging system, at least it doesn’t in the current firmware release.

Learning AprilTags is similar to learning colors. You will want to be in Learn Multiple mode in most cases, so select that in the usual fashion.  Then aim the HuskyLens at the first AprilTag you want to learn and press the Learning key. You only need to be brief, as a tag is a simple two-dimensional object.

Continue to learn as many tags as you wish.  When you are done, press down on the selector wheel to end the learning process.

Now the HuskyLens will detect the learned AprilTags. Note that it is capable of detecting them at any angle, not just the one you trained it in.

Object Classification

In Object Classification mode, the HuskyLens will allow you to classify several objects. For example, you could categorize different types of fruit like apples, oranges, lemons, etc.

As usual, you’ll want to be in Learn Multiple mode to work with multiple objects, so select it in the standard fashion.

Once you have done that, aim the HuskyLens at the first object that you wish to categorize and hold down the Learning key. Continue to hold down the key and learn the object from every angle that you can.

After you have finished learning the object, release the key. As with the other modes, you’ll be prompted to click again to learn another object.  Continue and recognize as many objects as you like.

When you have finished learning objects,  press down upon the selector wheel to end the training process.

The HuskyLens will now recognize the objects you have classified.

HuskyLens with Arduino

While the HuskyLens is a pretty impressive performer in its own right, the real excitement comes with using it as an intelligent “edge computing” device for a microcontroller.

As you recall from the earlier discussion about the HuskyLens specifications, this device can communicate with a host microcontroller in two fashions – Serial UART and I2C.

We are going to interface an Arduino Uno with the HuskyLens using both methods, starting with the Serial UART.

HuskyLens Hookup – UART Mode

In Serial UART mode, the HuskyLens is connected to the Arduino Uno as follows:

You can use the cable that comes with the HuskyLens, along with four short M-M jumper wires, to make the connection to the Arduino directly.

Once you have everything wired up, connect the Arduino to your computer. But before you start programming, there is one more task you need to take care of.

Installing the HuskyLens Library

DFRobot has provided a library for the HuskyLens, it really makes working with this little wonder very simple. However, the Library is not on the Arduino Library Manager list, nor is it available in an easy to import ZIP file. Instead, you will need to install it manually.

It’s probably best NOT to have the Arduino IDE open while you do this (although you won’t hurt anything if you do), so if it’s open you may wish to close it before proceeding.

First, you’ll need to obtain the library from the DFRobot HuskyLens Arduino GitHub page. On that page you’ll see all the resources for the HuskyLens Arduino API.

You’ll also see a green Code button with an arrow in it. Click on the arrow, and a sub-menu will drop down. One of the selections in the sub-menu is Download ZIP, which is what you want to do.

This ZIP file contains the library, but the directory structure is incorrect for the Arduino IDEs Import Library function.  So we will need to do the installation ourselves.

After you download the ZIP file, open it up with your archive tool.  Now navigate within the unzipped folder, you will see the root folder is called HUSKYLENSArduinoMaster. Inside that folder you should see three files plus another folder called HUSKYLENS.  

Highlight the HUSKYLENS folder and extract it to your Arduino libraries folder.  You’ll find the libraries folder in the same arduino folder that your Arduino sketch folders are stored in, which in most operating systems is in your Home or User folder. 

Once you have finished extracting the files, you can close your extraction tool, then open the Arduino IDE. Navigate to the File/Examples sub-menu and scroll down into the section of Examples from Custom Libraries. You should see a HUSKYLENS entry, which contains some sample code.

Let’s run some of that code right now.

HuskyLens Arduino Demo – UART Mode

The first example sketch that we will look at is the one titled GETTING_STARED, which probably means “getting started”. It’s one of the example sketches found in the HUSKYLENS examples folder we just looked for.

This sketch illustrates how to connect to the HuskyLens and read data from it. The data can be face, object or line data. The results will be displayed in the serial monitor.

The sketch starts by including the required libraries, the HuskyLens library, and the SoftwareSerial library. We will be requiring SoftwareSerial as the Arduino Uno we are using only has one serial port.  The library will let us define a second one using software.

We then create objects for both the HuskyLens and our software-created serial port. Note that we pass the Arduino pins that we have used to connect to the HuskyLens to the SoftwareSerial object. If you’d like to use different pins for the connection, you would need to change these numbers.

In the Setup, we initialize both the hardware and software serial connections, the hardware one being for our Serial Monitor. 

We then connect the HuskyLens to the SoftwareSerial connection, displaying an error message on the Serial Monitor if the connection fails.  If it does fail you’ll want to check the wiring from the HuskyLens to the Arduino.

Assuming that we establish a connection, we jump into the Loop.

In the Loop, we start by checking several things:

• Do we have an established connection to the HuskyLens?
• Has the HuskyLens learned anything (i.e. an object or line)?
• Is one of those learned objects detected on the screen right now?

If the answer to all three questions is YES, then the code jumps into a while loop, reading the data and displaying it to the Serial Monitor.  If the data drops off, such as when the target object goes out of view, the Loop will repeat.

The data for the Serial Monitor is formatted by a function called printResult. It determines if the detected item is a line or a block (such as a face or an object) and formats the text appropriately.

Load the code onto the Arduino, set the HuskyLens in the desired mode and observe the serial monitor. I tried it in Face Recognition mode and the display gave me the position of the block that detected the face, along with the face ID number.

You can see how you could make use of this information in your own project code.

HuskyLens Arduino Hookup – I2C Mode

You can also use the HuskyLens in I2C mode. In order to do this, you’ll need to change your wiring slightly.

The VCC (5V) and Ground connections remain the same. The TX (green wire) is moved to A4, and the other wire (the blue RX wire) goes to pin A5. 

HuskyLens Arduino Demo – I2C Mode

The next sketch is also included in the HuskyLens examples that were installed along with the library. It is exactly the same sketch we looked at earlier, except this time we connect to the HuskyLens via I2C.

The only difference you’ll note in the code is that we are using the Wire library instead of the SoftwareSerial library. The authors of the sketch didn’t declare the Wire library, but as it is a built-in library, it isn’t really necessary.

Note how the HuskyLens is bound to the wire object in its “begin” statement.

Once again, you need to put your HuskyLens into the desired mode and load up the sketch.  This time I did line tracking, and I used my finger as the line! In this mode, you get origin and target coordinates, as well as the line ID number. Great information if you’re designing a line tracking robot.

Modifying HuskyLens Text with Arduino

When recognizing faces or colors, it is a bit awkward to identify your targets by ID number. A proper name would be preferable.

By writing data to the HuskyLens you can change the text displayed on the device’s screen.

Before we start the next experiment, you’ll need to train the HuskyLens to recognize one or more objects or people.  Once you do, you’ll need to note the ID numbers assigned to each object or person.

In my case, I trained it for four faces, which came up as ID numbers 1 through 4.

Now I’ll run some code to put names to those faces.  My HuskyLens is still connected using I2C.

Arduino Name-changing Sketch

The sketch to change the four ID numbers into names is as follows:

We start by including both the HuskyLens and Wire libraries.  We then define an object to represent our HuskyLens.

I also defined a function to actually change the name, we’ll get to it in a moment, but first we’ll look at the Setup.

In Setup, we start the Serial Monitor, which is only used for debug purposes. We also start the I2C connection and bind to the HuskyLens object.

After a short delay to allow the HuskyLens to stabilize, we call the name changing function four times, once for each name we wish to change. Let’s look at that function now.

The function, setNewName, simply calls an existing function that is part of the HuskyLens library. The HuskyLens setCustomName method does exactly what we want, it takes a name and an ID number as its input parameters.

All that the function setNewName does is call setCustomName and verify that it succeeded. If it didn’t, then it is called again after a short delay.

I found this to be more reliable than simply calling setCustomName and adding a short delay, but you can experiment and see which one works for you.

You’ll probably want to modify the names I used in the sketch for your own situation. Once you do that, load the sketch to the Arduino while it is connected to the HuskyLens.

You should see the names change on the HuskyLens screen.

Using the SD Card

The HuskyLens has a microSD card slot on its face, the card inserts with the contacts down. 

You can use the microSD card to save photos, although the resolution is limited to the 320×240 size that you see on the HuskyLens screen.  You can also save snapshots, which are photos with the screen text as well. 

The SD card can also be used to save models you have learned. So, for example, if you teach the HuskyLens to recognize a face, you can save it to the SD card and then call it up later.

To save a model, for example some faces you have learned, press down on the selector wheel so that you activate the bottom menu. Scroll to the end of the menu, where you will see an Export to SD selection.

When you select this, you’ll get a scroll menu that goes from 0 to 5. This allows you to assign a number to your saved data, so that you can save more than one training set. 

After you have saved the data, you can recall it anytime. If you want to give that a try, then save some learned data and then go into General Settings and do a Factory Reset. This will wipe all the saved models from your HuskyLens.

Then go back into Face Recognition (assuming that it was face data you saved) and activate the bottom menu. Scroll over to Import from SD and select it. You’ll be prompted for the number of the saved settings, choose the same one you saved. 

After importing, you will need to reset the HuskyLens (power it off and back on). After that, try out the Face Recognition. You should see your saved data models are back now.

You can save data for virtually all the HuskyLens modes.

Conclusion

I was pretty impressed by the HuskyLens, this low-cost device really worked pretty well. It will certainly open up a world of new and exciting projects that can include machine vision.

What is particularly fascinating about this device is that using it,  you can start building AI projects without requiring any AI training.  

I’ve actually started on one of those projects myself, a small robot based upon the HuskyLens 9and a few other technologies). I’ll be presenting that soon, so keep a lookout for it here on the DroneBot Workshop website.