Autonomous SYMBIOSIS System Monitors Schools of Fish in Real-Time

Researchers in the water with the SYMBIOSIS unit. (Credit: © IMDEA Networks Institute | SYMBIOSIS Project, via communication from source)

A new autonomous system, which monitors schools of fish in real-time using acoustic and optical technologies, has been developed to provide scientists with better information about marine fish stocks anywhere in the world’s oceans. Created by researchers from the University of Haifa and two IMDEA Networks Institute teams, the system is called SYMBIOSIS.

Monitoring marine communities without human intervention

Researchers who collaborated to develop SYMBIOSIS were responding to several knotty problems in marine fauna monitoring.

“We observed that currently available approaches for evaluating marine biota can only provide limited solutions, often confined to benign, or specific environments,” explains Dr. Paolo Casari, the project’s scientific manager and IMDEA principal investigator. “We wanted to advance beyond the state of the art by developing a prototype of a non-invasive system for monitoring deep and coastal waters for fish stock, which would work over a broad range of depths and distances from shore.”

The SYMBIOSIS system’s real-time monitoring capability will allow a more evidence-based approach to fishing policy development and enhance our ability as humans to protect the marine environment, as its optical/acoustic approach helps to prevent false alarms.

“The SYMBIOSIS platform is a hybrid system that integrates acoustic components (a projector and several receiving hydrophones) together with optical components (optical cameras and strobes) and with the hardware and control logic required to run the system and process the obtained data,” details Dr. Casari. “A long-range WiFi link will enable communications to nearby control stations or ships.”

Current fish monitoring methods include deploying research boats with crews within limited areas. SYMBIOSIS is designed to cover far more area without the need for a human crew—the chief expense during the assessment of marine stocks.

Costs are often compounded because monitoring fishing stocks are tricky. The underwater environment is extremely diverse and challenges acoustic and optical systems in several ways.

The SYMBIOSIS team, with Dr. Casari the first on the left. (Credit: © IMDEA Networks Institute | SYMBIOSIS Project, https://www.networks.imdea.org/whats-new/news/2018/innovative-autonomous-system-identifying-schools-fish)

“For example, acoustic imaging techniques produce a very narrow beam to detect objects located directly below a surveying vessel, and does not fit the task of monitoring the marine environment,” remarks Dr. Casari. “Moreover, there are no reliable commercial methods for the classification and biomass evaluation of fish, and the capabilities of the existing methods for fish detection are highly limited to simple seabed environments.”

In fact, Dr. Casari explains that current detection capabilities involve many false alarms in non-homogeneous seabed environments, including highlights, sand ripples, shadows, seagrass, and boulders. Furthermore, research is often performed manually.

“As a result, current surveys are limited in space and require a long analysis process,” concludes Dr. Casari. “In this context, our goal is to develop a reliable prototype to autonomously detect, classify, and enumerate marine species. This may not only improve the efficiency of environmental monitoring tasks but will also open the way for long-term and long-range surveys.”

Using deep learning to identify slippery fish

SYMBIOSIS operates with the help of an algorithm and image recognition technology.

“Periodically, the SYMBIOSIS platform will run a very energy-efficient algorithm to detect the presence of one of the fish species that the project focuses on [six large, high-demand fish species, including two species of tuna; scad (a type of mackerel); Atlantic mackerel; mahi-mahi; and swordfish],” Dr. Casari describes. “Once the presence detection phase is complete, the same acoustic system will be used to track the detected fish as it approaches or swims around the platform. If the fish finally enters the range of the optical cameras mounted on the platform, those cameras facing the approaching fish will be activated to capture images. Such images will help validate the acoustic detection through a computerized image recognition system, inspired to those employed to recognize faces and objects. If needed, relevant data can be recovered from the platform periodically, or transferred wirelessly to a nearby station through the long-range WiFi link.”

A graphic demonstration of the Symbiosis system. (Credit: © IMDEA Networks Institute | SYMBIOSIS Project, https://www.eurekalert.org/multimedia/pub/173530.php?from=397624)

This process does more than prevent false alarms and provide more accurate results; it is also designed to be less invasive and more friendly to the environment as it monitors the marine environment within a radius of one kilometer.

“SYMBIOSIS will employ active detection through low-energy acoustic signals in order to avoid having an excessive impact on marine fauna,” remarks Dr. Casari. “However, this detection phase will be activated only if a preliminary detection step determined that some fish of interest is actually close to the platform. This avoids the continuous pinging of the underwater environment. The cameras that are employed to validate the detections and their artificial illumination aids are only activated for extremely short time periods, only if fishes of interest are detected close to the camera range, and only towards their direction of arrival. As a result, underwater lighting pollution is greatly reduced. Also notably, our system makes no use of fishing nets and requires no tagging.”

The idea to impact the marine ecosystem minimally is as appealing as the need for monitoring is pressing. Overfishing is out of control, and some estimates indicate a total collapse of world fish stocks by 2048 without intervention into the overfishing problem.

Localizing particular species using acoustics is a tremendous challenge, not least because high energy requirements make it difficult to maintain a cost-effective monitoring process. SYMBIOSIS solves this problem in part, striking a balance between accuracy, complexity, and cost.

“The system activation procedure is very hierarchical,” comments Dr. Casari. “First comes a coarse acoustic detection, and upon a positive outcome, active detection and tracking follow. Images are captured and processed only if the acoustic components tell that fishes of interest come within the camera range. Fish recognition will be done using deep learning networks that will be trained offline. These ‘just-in-time’ features of the SYMBIOSIS platform make it a low-energy solution for underwater environmental monitoring.”

The system classifies fish based on their usual movement characteristics and speed using acoustic sensors. Those sensors also calculate the total biomass of nearby fish and the size of the fish in range. The optical system is activated only after the acoustic system identifies one of its six target species. The identification algorithms process high-quality images using deep learning.

The team prepares to deploy SYMBIOSIS during a recent research test run. (Credit: © IMDEA Networks Institute | SYMBIOSIS Project, via communication from source)

“The optical part works as a detection validation for the acoustic part,” adds Dr. Casari. “Therefore, in contrast to systems that are only acoustic or only optical, our system merges these two worlds, which results in a minimization of false alarms and improves data quality for the scientists that will eventually use the data.”

“Buoying” fish stock monitoring with new technologies

Once it is deployed, all components of SYMBIOSIS will mount to a buoy under the water’s surface.

“There will be a set of six cameras covering the whole horizontal plane at 10 meters of depth,” states Dr. Casari. “The acoustic detection system will be installed below it, and will be followed by one more set of six cameras at a lower depth, to provide diversity and reduce the impact of obstructions in the cameras’ fields of view.”

Waterproof housing will host the boards that process images. Finally, Dr. Casari explains, the buoy will host a communication system that will be used to observe the behavior of the equipment during the development stage, and will double as a long haul for relevant data at a later stage of the project, when the system will work autonomously and communicate relevant data to shore or nearby research vessels.

While SYMBIOSIS has been developed for tracking six species in danger of over-exploitation in particular, it will almost certainly be deployed in other ways.

“It can in principle be configured to work with any other species, provided that enough knowledge of their acoustic characterization exists (to help interpret the signals received by the hydrophones and make accurate acoustic detections), and provided that a sufficient number of images are available in realistic conditions (to train the automatic image recognition algorithms),” adds Dr. Casari.

Team members prepare to deploy the unit on the mooring platform. (Credit: © IMDEA Networks Institute | SYMBIOSIS Project, via communication from source)

Furthermore, the SYMBIOSIS system is merely a prototype. For Dr. Casari and the team, the plan is to use this prototype as a starting point. From that point, they hope to extend the capabilities of autonomous underwater monitoring technologies in multiple ways, including further reduction of energy consumption; miniaturization of the system; and modification of the mooring platform aboard different vessels to promote a more mobile use of the system.

Dr. Casari and the team also need help and input.

“People can get involved in our effort!” remarks Dr. Casari. “There is a distinctive issue with image recognition algorithms based on machine learning: they need high-quality data for training. Considering that training requires hundreds of thousands of images, it can take a huge effort to correctly classify all images as ‘useful’ for training.”

Therefore, the team is setting up a crowdsourcing system that will help the public at large get involved in this process.

“People will get the chance to learn more about marine biology, and they’ll be able to use this knowledge to help us classify images correctly,” states Dr. Casari. “As a result, the SYMBIOSIS system will be more accurate. A website will be out in the next months and will be advertised on our web page.”

Top image: Researchers in the water with the SYMBIOSIS unit. (Credit: © IMDEA Networks Institute | SYMBIOSIS Project, via communication from source)