Accelerating Ukraine Intelligence Evaluation with Laptop Imaginative and prescient on Artificial Aperture Radar Imagery

0/5 No votes

Report this app



Determine 1: Airmass measurements over Ukraine from February 18, 2022 – March 01, 2022 from the SEVIRI instrument. Knowledge accessed through the EUMETSAT Viewer.

Satellite tv for pc imagery is a vital supply of knowledge throughout the present invasion of Ukraine. Navy strategists, journalists, and researchers use this imagery to make choices, unveil violations of worldwide agreements, and inform the general public of the stark realities of battle. With Ukraine experiencing a considerable amount of cloud cowl and assaults usually occuring throughout night-time, many types of satellite tv for pc imagery are hindered from seeing the bottom. Artificial aperture radar imagery penetrates cloud cowl, however requires particular coaching to interpret. Automating this tedious process would allow real-time insights, however present pc imaginative and prescient strategies developed on typical RGB imagery don’t correctly account for the phenomenology of SAR. This results in suboptimal efficiency on this vital modality. Enhancing the entry to and availability of SAR-specific strategies, codebases, datasets, and pretrained fashions will profit intelligence businesses, researchers, and journalists alike throughout this vital time for Ukraine.

On this put up, we current a baseline methodology and pretrained fashions that allow the interchangeable use of RGB and SAR for downstream classification, semantic segmentation, and alter detection pipelines.


Satellite tv for pc imagery is a vital supply of knowledge throughout the present invasion of Ukraine. Navy strategists, journalists, and researchers use this imagery to make choices, unveil violations of worldwide agreements, and inform the general public of the stark realities of battle. With Ukraine experiencing a considerable amount of cloud cowl and assaults usually occuring throughout night-time, many types of satellite tv for pc imagery are hindered from seeing the bottom.

As such, there was an elevated reliance on a type of satellite tv for pc imagery often called Artificial Aperture Radar (SAR) to supply visibility in all types of antagonistic environmental circumstances. With its non-standard and unintuitive phenomenology, a whole bunch of specifically educated SAR analysts have been manually combing by way of billions of pixels making an attempt to supply actionable intelligence in real-time.

Automating this tedious process allows real-time intelligence and democratizes SAR evaluation. Nonetheless, present pc imaginative and prescient strategies developed on typical RGB imagery don’t correctly account for the phenomenology of SAR, resulting in suboptimal efficiency on this vital modality. Enhancing the entry to and availability of SAR-specific strategies, codebases, datasets, and pretrained fashions will profit intelligence businesses, researchers, and journalists alike throughout this vital time for Ukraine.

On this put up, we current a baseline methodology and pretrained fashions that allow the interchangeable use of RGB and SAR for downstream classification, semantic segmentation, and alter detection pipelines. We’re engaged on transitioning these fashions to our US authorities and NGO companions to be used throughout this disaster, and we encourage others within the Laptop Imaginative and prescient neighborhood to contribute to this vital analysis space.


We stay in a quickly altering world, one which experiences pure disasters, civic upheaval, battle, and all types of chaotic occasions which depart unpredictable—and infrequently everlasting—marks on the face of the planet. Understanding this variation has traditionally been troublesome. Surveyors have been despatched out to discover our new actuality, and their distributed findings have been usually noisily built-in right into a supply of actuality. Sustaining a relentless state of vigilance has been a purpose of mankind since we have been capable of conceive such a thought, all the way in which from when Nadar took the primary aerial {photograph} to when Sputnik 1’s radio indicators have been used to research the ionosphere.

Vigilance, or to the French, surveillance, has been part of human historical past for millenia. As with every device, it has been a double-edged sword. Traditionally, surveillance with out checks and balances has been detrimental to society. Conversely, the correct and accountable surveillance has allowed us to be taught deep truths about our world which have resulted in advances within the scientific and humanitarian domains. With the quantity of satellites in orbit at the moment, our understanding of the setting is up to date virtually each day. We’ve got quickly transitioned from having little or no info to now having extra knowledge than we are able to meaningfully extract data from. Storing this info, not to mention understanding, is an engineering problem that’s of rising urgency.

Machine Studying and Distant Sensing

With a whole bunch of terabytes of information being downlinked from satellites to knowledge facilities each day, gaining data and actionable insights from that knowledge with guide processing has already turn into an unimaginable process. Essentially the most extensively used type of distant sensing knowledge is electro-optical (EO) satellite tv for pc imagery. EO imagery is commonplace—anybody who has used Google Maps or related mapping software program has interacted with EO satellite tv for pc imagery.

Machine studying (ML) on EO imagery is utilized in all kinds of scientific and business purposes. From bettering precipitation predictions, analyzing human slavery by figuring out brick kilns, to classifying total cities to enhance site visitors routing, the outputs of ML on EO imagery have been built-in into virtually each side of human society.

Determine 2: VHR EO imagery over the Kyiv area as acquired by Maxar on February 28, 2022.

Generally used satellite tv for pc constellations for EO imagery embody the Landsat collection of satellites operated by america Geological Survey and the Copernicus Sentinel-2 constellation operated by the European Area Company. These constellations present imagery at resolutions between 10-60 meters which is nice sufficient for a lot of use instances, however preclude the remark of finer particulars.

The Introduction of Very Excessive Decision, Business Electro-Optical Satellite tv for pc Imagery

Over the previous couple of years, very excessive decision (VHR) EO imagery has been made out there by way of quite a lot of business sources. Starting from between 0.3 – 2.0 meter decision, firms similar to Planet, Maxar, Airbus, and others are offering extraordinarily exact imagery with excessive revisit charges, imaging your complete planet each day.

Determine 3: An instance of Maxar VHR EO imagery exhibiting floating manufacturing, storage and off-loading models and a tanker.

The elevated decision offered by VHR imagery allows a litany of downstream use instances. Erosion will be detected at finer scales, and the constructing injury will be categorized after pure disasters.

Machine studying strategies have needed to adapt in response to VHR satellite tv for pc imagery. With an elevated acuity, the quantity of pixels and the quantity of lessons that may be discerned has elevated by orders of magnitude. Laptop imaginative and prescient analysis has responded by decreasing the computational value to be taught environment friendly illustration of satellite tv for pc imagery, creating strategies to alleviate the elevated burden on labelers, and even engineering giant software program frameworks to permit pc imaginative and prescient practitioners to deal with this considerable supply of images.

Usually, present pc imaginative and prescient strategies on different, non-aerial RGB imagery switch very effectively to satellite tv for pc imagery. This has allowed business VHR imagery to be instantly helpful with extremely correct outcomes.

The Drawback with Electro-Optical Imagery

For extremely turbulent and dangerous conditions similar to battle and pure disasters, having fixed, dependable entry to the Earth is paramount. Sadly, EO imagery can’t resolve all of our surveillance wants. EO can solely detect mild sources throughout daytime, and because it seems, practically 2/3rds of the Earth is roofed by clouds at any given time. Except you care about clouds, this blockage of the floor of the planet is problematic when understanding what occurs on the bottom is of vital significance. Machine studying strategies try to sidestep this drawback by predicting what the world would appear to be with out clouds. Nonetheless, the lack of info is essentially irrecoverable.

Artificial Aperture Radar Imagery

Artificial aperture radar (SAR) imagery is an energetic type of distant sensing during which a satellite tv for pc transmits pulses of microwave radar waves right down to the floor of the Earth. These radar waves mirror off the bottom and any objects on it and are returned again to the satellite tv for pc. By processing these pulses over time and house, a SAR picture is fashioned the place every pixel is the superposition of various radar scatters.

Radar waves penetrate clouds, and for the reason that satellite tv for pc is actively producing the radar waves, it illuminates the floor of the Earth even throughout the night time. Artificial aperture radar has all kinds of makes use of, getting used to estimate the roughness of the Earth, mapping the extent of flooding over giant areas, and to detect the presence of unlawful fishing vessels in protected waters.

There are a number of SAR satellite tv for pc constellations in operation in the intervening time. The Copernicus Sentinel-1 constellation supplies imagery to the general public at giant with resolutions starting from 10 – 80 meters (10 meter imagery being the most typical. Most business SAR suppliers, similar to ICEYE and Capella Area, present imagery right down to 0.5 meter decision. In upcoming launches, different business distributors purpose to supply SAR imagery with sub-0.5 meter decision with excessive revisit charges as satellite tv for pc constellations develop and authorities rules evolve.

Determine 4: A VHR SAR picture offered by Capella Area over the Ukraine-Belarus border.

The Wacky World of Artificial Aperture Radar Imagery

Whereas SAR imagery, at a fast look, could look similar to EO imagery, the underlying physics is kind of completely different, which ends up in many fascinating results within the imagery product which will be counterintuitive and incompatible with trendy pc imaginative and prescient. Three widespread results are termed polarization, layover, and multi-path results.

Radar antennas on SAR satellites usually transmit polarized radar waves. The path of polarization is the orientation of the wave’s electrical subject. Objects on the bottom exhibit completely different responses to the completely different polarizations of radar waves. Subsequently, SAR satellites usually function in twin or quad-polarization modes, broadcasting horizontally (H) or vertically (V) polarized waves and studying both polarization again, leading to HH, HV, VH, and VV bands. You may distinction this with RGB bands in EO imagery, however the basic physics are completely different.

Determine 5: Distinction between VH (left) and VV (proper) polarizations over the identical area in Dnipro, Ukraine from Sentinel-1 radiometric terrain corrected imagery. As seen right here, the radar returns in corresponding native areas will be completely different.

Layover is an impact during which radar beams attain the highest of a construction earlier than they attain the underside, ensuing within the prime of the item being offered as overlapping with the underside. This occurs when objects are significantly tall. Visually, tall buildings seem as if they’re laying on their facet, whereas mountains can have their peaks intersecting with their bases.

Determine 6: Instance of layover in Capella’s VHR SAR imagery. The higher portion of the stadium is intersecting, seemingly, with the parking zone behind it.

Multi-path results happen when radar waves mirror off of objects on the bottom and incur a number of bounces earlier than returning to the SAR sensor. Multi-path results end in objects showing within the imagery in numerous transformations within the ensuing picture. This impact will be seen all over the place in SAR imagery, however is especially noticeable in city areas, forests, and different dense environments.

Determine 7: Instance of a multi-path impact on a bridge from indirect SAR imagery.

Current pc imaginative and prescient strategies which are constructed on conventional RGB imagery should not constructed with these results in thoughts. Object detectors educated on EO satellite tv for pc imagery assume {that a} distinctive object will solely seem as soon as, or that the item will seem comparatively related in several contexts, reasonably than probably mirrored or scattered or interwoven with surrounding objects. The very nature of occlusion and the imaginative and prescient rules underlying the assumptions of occlusion in EO imagery don’t switch to SAR. Taken collectively, present pc imaginative and prescient strategies can switch to SAR imagery, however with diminished efficiency and a set of systematic errors that may be addressed by way of SAR-specific methodology.

Laptop Imaginative and prescient on SAR Imagery for Ukraine

Imagery analysts are at the moment counting on each EO and SAR imagery the place out there over Ukraine. When EO imagery is out there, present pc imaginative and prescient tooling constructed for that modality is used to expedite the method of intelligence gathering. Nonetheless, when solely SAR imagery is out there, these toolchains can’t be used. Imagery analysts must resort to guide evaluation which is time consuming and will be liable to errors. This subject is being explored by another establishments internationally, nonetheless, it nonetheless stays an understudied space with respect to the quantity of information out there.

At Berkeley AI Analysis, we have now created an preliminary set of strategies and fashions which have realized sturdy representations for RGB, SAR, and co-registered RGB + SAR imagery from the publicly launched BigEarthNet-MM dataset and the info from Capella’s Open Knowledge, which consists of each RGB and SAR imagery. As such, utilizing our fashions, imagery analysts are capable of interchangeably use co-registered RGB or SAR imagery (or each, when out there) for downstream duties similar to picture classification, semantic segmentation, object detection, or change detection.

On condition that SAR is a phenomenologically completely different knowledge supply than EO imagery, we have now discovered that the Imaginative and prescient Transformer (ViT) is a very efficient structure for illustration studying with SAR because it removes the dimensions and shift invariant inductive biases constructed into convolutional neural networks. Our prime performing methodology, MAERS, for illustration studying on RGB, SAR, and co-registered RGB + SAR builds upon the Masked Autoencoder (MAE) just lately launched by He et. al., the place the community learns to encode the enter knowledge by taking a masked model of the info as enter, encoding the info, after which studying to decode the info in such a manner that it reconstructs the unmasked enter knowledge.

Opposite to widespread lessons of contrastive studying strategies, the MAE doesn’t presuppose sure augmentation invariances within the knowledge that could be incorrect for SAR options. As a substitute, it solely depends on reconstructing the unique enter, which is agnostic to RGB, SAR, or co-registered modalities. As proven in Determine 8, MAERS additional extends MAE by studying unbiased enter projection layers for RGB, SAR, and RGB+SAR channels, encoding the output of those projected layers utilizing a shared ViT, after which decoding to the RGB, SAR, or RGB+SAR channels utilizing unbiased output projection layers. The enter projection layers and shared ViT can then be transferred to downstream duties, similar to object detection or change detection, the place the enter encoder can then take RGB, SAR, or RGB+SAR as enter.

Determine 8: (prime) A visualization of MAERS to be taught an optionally joint illustration and encoder that can be utilized for a (backside) downstream process, similar to object detection on both, or each, modalities.

Studying representations for RGB, SAR, and co-registered modalities can profit a spread of downstream duties, similar to content-based picture retrieval, classification, segmentation, and detection. To reveal the effectiveness of our realized representations, we carry out experiments on the well-established benchmarks of 1) multi-label classification of co-registered EO and SAR scenes from the BigEarthNet-MM dataset, and a pair of) semantic segmentation on the VHR EO and SAR SpaceNet 6 dataset.

Multi-Label Classification on BigEarth-MM

Determine 9: (left) co-registered Sentinel-2 EO and Sentinel-1 SAR imagery are patchified and used to carry out a multi-label classification process as specified by the BigEarth-MM problem. A linear layer is added to our multi-modal encoder after which fine-tuned end-to-end.

MAERS is initialized with a set of ImageNet weights for a ViT-Base encoder, adopted by pretraining on the BigEarthNet-MM dataset for 20 epochs with RGB, SAR, and RGB+SAR imagery. We append a single linear layer to the MAERS encoder and be taught the multi-label classification process by fine-tuning your complete mannequin for 20 epochs (linear probe experimental outcomes present related outcomes, as we’ll present in our upcoming paper). Our outcomes are proven in Desk 1. MAERS with fine-tuning outperforms the perfect RGB+SAR outcomes as offered within the BigEarthNet-MM paper, and present that adapting the State-of-the-Artwork MAE structure for illustration studying for RGB, SAR, and RGB+SAR enter modalities results in State-of-the-Artwork outcomes.

Desk 1: Reported per-class F2 scores on the take a look at set of BigEarthNet-MM.

Semantic Segmentation on VHR EO and SAR SpaceNet 6

We additional experimented with switch studying for a well timed process that can help imagery analysts aiming to grasp the destruction in Ukraine: semantic segmentation of buildings footprints, which is a precursor process to performing constructing injury evaluation. Constructing injury evaluation is of direct curiosity to authorities officers, journalists, and human rights organizations aiming to grasp the scope and severity of Russia’s assaults towards infrastructure and civilian populations.

Determine 10: Instance of constructing SAR-based MAERS segmentation taken from SpaceNet6, the place the picture on the left exhibits the RGB picture, and the picture on the best exhibits the SAR picture with overlaid segmentation outcomes. The SAR picture is displayed in false shade with VV, VH, and VV/VH bands.

For this experiment, we used the SpaceNet 6 dataset as an open and public benchmark for instance the effectiveness of our realized representations for constructing footprint detection with VHR SAR from Capella Area. We used this encoder in tandem with the UperNet structure for semantic segmentation. Determine 9 exhibits the IoU efficiency of segmenting constructing footprints in a held-out validation element of the SpaceNet 6 with solely SAR enter imagery, on a segmentation mannequin that was educated to make use of both SAR or RGB imagery. The MAERS pretrained mannequin results in a ~13 level enchancment in comparison with coaching the RGB+SAR mannequin from scratch or adapting ImageNet weights with the very same structure.

This demonstrates that MAERS can be taught sturdy RGB+SAR representations that enable a practitioner to make use of EO or SAR imagery interchangeably to perform downstream duties. It is very important be aware that the phenomenology of SAR imagery just isn’t absolutely conducive for constructing segmentation and that utilizing EO imagery for this process results in IoU scores > 90. This leaves a considerable hole but to be lined by SAR strategies, one thing we hope to cowl in our following paper. Nonetheless, getting this efficiency out of SAR is important when environmental circumstances should not conducive to EO imagery seize.

Determine 11: Constructing segmentation IoU on the SpaceNet 6 Problem, utilizing an UperNet segmentation mannequin with a ViT spine. MAERS pretraining results in ~13 level achieve in IoU efficiency in comparison with coaching from scratch or adapting ImageNet pretrained weights.

These outcomes are preliminary, however compelling. We’ll comply with up this effort with a publication with an in depth set of experiments and benchmarks. Moreover, we’ll help within the transition of our fashions to our humanitarian companions to allow them to carry out change detection over residential and different civilian areas to allow higher monitoring of battle crimes being dedicated in Ukraine.

These fashions are created with the purpose of accelerating the efficacy of organizations concerned in humanitarian missions which are maintaining a watchful eye on the battle in Ukraine. Nonetheless, as with all expertise, it’s our duty to grasp how this expertise may very well be misused. Subsequently, we have now designed these fashions with enter from companions who carry out intelligence and imagery evaluation in humanitarian settings. By taking into consideration their ideas, feedback, and critiques, we’re releasing a functionality we’re assured will likely be used for the great of humanity and with processes which dictate their protected and accountable use.

Name to Motion

As residents of free democracies who develop applied sciences which assist us make sense of the sophisticated, chaotic, and counter-intuitive world that we stay in, we have now a duty to behave when acts of injustice happen. Our colleagues and pals in Ukraine are going through excessive uncertainties and hazard. We possess expertise within the cyber area that may help within the combat towards Russian forces. By focusing our time and efforts, whether or not that be by way of focused analysis or volunteering our time in serving to preserve monitor of processing instances at border crossings, we are able to make a small dent in an in any other case troublesome scenario.

We urge our fellow pc scientists to associate with authorities and humanitarian organizations and take heed to their wants as troublesome instances persist. Easy issues could make giant variations.

Mannequin and Weights

The fashions should not being made publicly accessible presently. We’re releasing our fashions to certified researchers and companions by way of this kind. Full distribution will comply with as soon as we have now accomplished a radical evaluation of our fashions.


Thanks to Gen. Steve Butow and Dr. Nirav Patel on the Division of Protection’s Protection Innovation Unit for reviewing this put up and offering their experience on the way forward for business SAR constellations.


Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.