Publications

In recent years, deep learning has emerged as the dominant approach for flood mapping from remote sensing satellite imagery. While new flood segmentation models are increasingly being proposed, most of these works focus on advancing architectures trained on single datasets. Therefore, these studies overlook the intrinsic limitations and biases of the available training and evaluation data. This often leads to poor generalization and overconfident predictions when these models are used in real-world scenarios. To address this gap, the objective of this work is twofold. First, we train and evaluate flood segmentation models on five publicly available datasets including data from Sentinel-1, Sentinel-2, and both SAR and multispectral modalities. Our findings reveal that models achieving high detection accuracy on a single dataset (intra-dataset validation) do not necessarily generalize well to unseen datasets. In contrast, models trained on more diverse samples from multiple datasets demonstrate greater robustness and generalization ability. Furthermore, we present a dual-stream multimodal architecture that can be independently trained and tested on both single-modality and dual-modality datasets. This enables the integration of all the diversity and richness of available data into a single unified framework. The results emphasize the need for a more comprehensive validation using diverse and well-designed datasets, particularly for multimodal approaches. If not adequately addressed, the shortcomings of current datasets can significantly limit the potential of deep learning-based flood mapping approaches.

Mitigating methane emissions is the fastest way to stop global warming in the short-term and buy humanity time to decarbonise. Despite the demonstrated ability of remote sensing instruments to detect methane plumes, no system has been available to routinely monitor and act on these events. We present MARS-S2L, an automated AI-driven methane emitter monitoring system for Sentinel-2 and Landsat satellite imagery deployed operationally at the United Nations Environment Programme's International Methane Emissions Observatory. We compile a global dataset of thousands of super-emission events for training and evaluation, demonstrating that MARS-S2L can skillfully monitor emissions in a diverse range of regions globally, providing a 216\% improvement in mean average precision over a current state-of-the-art detection method. Running this system operationally for six months has yielded 457 near-real-time detections in 22 different countries of which 62 have already been used to provide formal notifications to governments and stakeholders.

Code

Web page

Nano and microsatellites have expanded the acquisition of satellite images with higher spatial, temporal, and spectral resolutions. Nevertheless, downlinking all this data to the ground for processing becomes challenging as the amount of remote sensing data rises. Custom onboard algorithms are designed to make real-time decisions and to prioritize and reduce the amount of data transmitted to the ground. However, these onboard algorithms frequently require cloud-free bottom-of-atmosphere surface reflectance (SR) estimations as inputs to operate. In this context, this article presents the data transformations and autocalibration for Sentinel-2 (S-2) network (DTACSNet), an onboard cloud detection and atmospheric correction processor based on lightweight convolutional neural networks. DTACSNet provides cloud and cloud shadow masks and SR estimates 10× faster than the operational S-2 L2A processor in dedicated space-tested hardware: 7 mins versus 1 h for a 10 980 × 10 980 scene. The DTACSNet cloud masking, based on a lightweight neural network, obtains the highest F2-score (0.81), followed by the state-of-the-art KappaMask (0.74), Fmask (0.72), and Sen2Cor v.2.8 (0.51) algorithms. In addition, validation results on independent datasets show that DTACSNet can efficiently replicate Sen2Cor SR estimates, reporting a competitive accuracy with differences below 2\%.

Code

Web page

Detecting and screening clouds is the first step in most optical remote sensing analyses. Cloud formation is diverse, presenting many shapes, thicknesses, and altitudes. This variety poses a significant challenge to the development of effective cloud detection algorithms, as most datasets lack an unbiased representation. To address this issue, we have built CloudSEN12+, a significant expansion of the CloudSEN12 dataset. This new dataset doubles the expert-labeled annotations, making it the largest cloud and cloud shadow detection dataset for Sentinel-2 imagery up to date. We have carefully reviewed and refined our previous annotations to ensure maximum trustworthiness. We expect CloudSEN12+ will be a valuable resource for the cloud detection research community.

Code

We present a deep learning model, CH4Net, for automated monitoring of methane super-emitters from Sentinel-2 data. When trained on images of 23 methane super-emitter locations from 2017–2020 and evaluated on images from 2021, this model detects 84 \% of methane plumes compared with 24 \% of plumes for a state-of-the-art baseline while maintaining a similar false positive rate. We present an in-depth analysis of CH4Net over the complete dataset and at each individual super-emitter site. In addition to the CH4Net model, we compile and make open source a hand-annotated training dataset consisting of 925 methane plume masks as a machine learning baseline to drive further research in this field.

Code

Web page

Floods are among the most destructive extreme events that exist, being the main cause of people affected by natural disasters. In the near future, estimated flood intensity and frequency are projected to increase. In this context, automatic and accurate satellite-derived flood maps are key for fast emergency response and damage assessment. However, current approaches for operational flood mapping present limitations due to cloud coverage on acquired satellite images, the accuracy of flood detection, and the generalization of methods across different geographies. In this work, a machine learning framework for operational flood mapping from optical satellite images addressing these problems is presented. It is based on a clouds-aware segmentation model trained in an extended version of the WorldFloods dataset. The model produces accurate and fast water segmentation masks even in areas covered by semitransparent clouds, increasing the coverage for emergency response scenarios. The proposed approach can be applied to both Sentinel-2 and Landsat 8/9 data, which enables a much higher revisit of the damaged region, also key for operational purposes. Detection accuracy and generalization of proposed model is carefully evaluated in a novel global dataset composed of manually labeled flood maps. We provide evidence of better performance than current operational methods based on thresholding spectral indices. Moreover, we demonstrate the applicability of our pipeline to map recent large flood events that occurred in Pakistan, between June and September 2022, and in Australia, between February and April 2022. Finally, the high-resolution (10-30m) flood extent maps are intersected with other high-resolution layers of cropland, building delineations, and population density. Using this workflow, we estimated that approximately 10 million people were affected and 700k buildings and 25,000 km\$\${\textasciicircum}2\$\$of cropland were flooded in 2022 Pakistan floods.

Code

Methane is the second most important greenhouse gas contributor to climate change; at the same time its reduction has been denoted as one of the fastest pathways to preventing temperature growth due to its short atmospheric lifetime. In particular, the mitigation of active point-sources associated with the fossil fuel industry has a strong and cost-effective mitigation potential. Detection of methane plumes in remote sensing data is possible, but the existing approaches exhibit high false positive rates and need manual intervention. Machine learning research in this area is limited due to the lack of large real-world annotated datasets. In this work, we are publicly releasing a machine learning ready dataset with manually refined annotation of methane plumes. We present labelled hyperspectral data from the AVIRIS-NG sensor and provide simulated multispectral WorldView-3 views of the same data to allow for model benchmarking across hyperspectral and multispectral sensors. We propose sensor agnostic machine learning architectures, using classical methane enhancement products as input features. Our HyperSTARCOP model outperforms strong matched filter baseline by over 25\% in F1 score, while reducing its false positive rate per classified tile by over 41.83\%. Additionally, we demonstrate zero-shot generalisation of our trained model on data from the EMIT hyperspectral instrument, despite the differences in the spectral and spatial resolution between the two sensors: in an annotated subset of EMIT images HyperSTARCOP achieves a 40\% gain in F1 score over the baseline.

Web page

Cognitive cloud computing in space (3CS) describes a new frontier of space innovation powered by Artificial Intelligence, enabling an explosion of new applications in observing our planet and enabling deep space exploration. In this framework, machine learning (ML) payloads—isolated software capable of extracting high level information from onboard sensors—are key to accomplish this vision. In this work we demonstrate, in a satellite deployed in orbit, a ML payload called ‘WorldFloods’ that is able to send compressed flood maps from sensed images. In particular, we perform a set of experiments to: (1) compare different segmentation models on different processing variables critical for onboard deployment, (2) show that we can produce, onboard, vectorised polygons delineating the detected flood water from a full Sentinel-2 tile, (3) retrain the model with few images of the onboard sensor downlinked to Earth and (4) demonstrate that this new model can be uplinked to the satellite and run on new images acquired by its camera. Overall our work demonstrates that ML-based models deployed in orbit can be updated if new information is available, paving the way for agile integration of onboard and onground processing and “on the fly” continuous learning.

Code

Visualization

Video

High resolution remote sensing imagery is used in a broad range of tasks, including detection and classification of objects. High-resolution imagery is however expensive to obtain, while lower resolution imagery is often freely available and can be used for a range of social good applications. To that end, we curate a multi-spectral multi-image dataset for super-resolution of satellite images. We use PlanetScope imagery from the SpaceNet-7 challenge as the high resolution reference and multiple Sentinel-2 revisits of the same location as the low-resolution imagery. We present the first results of applying multi-image super-resolution (MISR) to multi-spectral remote sensing imagery. We, additionally, introduce a radiometric-consistency module into the MISR model to preserve the high radiometric resolution and quality of the Sentinel-2 sensor. We show that MISR is superior to single-image super-resolution (SISR) and other baselines on a range of image fidelity metrics. Furthermore, we present the first assessment of the utility of multi-image super-resolution on a semantic and instance segmentation – common remote sensing tasks – showing that utilizing multiple images results in better performance in these downstream tasks, but MISR pre-processing is non-essential.

Code

Visualization

Web page

Accurately characterizing clouds and their shadows is a long-standing problem in the Earth Observation community. Recent works showcase the necessity to improve cloud detection methods for imagery acquired by the Sentinel-2 satellites. However, the lack of consensus and transparency in existing reference datasets hampers the benchmarking of current cloud detection methods. Exploiting the analysis-ready data offered by the Copernicus program, we created CloudSEN12, a new multi-temporal global dataset to foster research in cloud and cloud shadow detection. CloudSEN12 has 49,400 image patches, including (1) Sentinel-2 level-1C and level-2A multi-spectral data, (2) Sentinel-1 synthetic aperture radar data, (3) auxiliary remote sensing products, (4) different hand-crafted annotations to label the presence of thick and thin clouds and cloud shadows, and (5) the results from eight state-of-the-art cloud detection algorithms. At present, CloudSEN12 exceeds all previous efforts in terms of annotation richness, scene variability, geographic distribution, metadata complexity, quality control, and number of samples.

Code

Applications such as disaster management enormously benefit from rapid availability of satellite observations. Traditionally, data analysis is performed on the ground after being transferred—downlinked—to a ground station. Constraints on the downlink capabilities, both in terms of data volume and timing, therefore heavily affect the response delay of any downstream application. In this paper, we introduce RaVÆn, a lightweight, unsupervised approach for change detection in satellite data based on Variational Auto-Encoders (VAEs), with the specific purpose of on-board deployment. RaVÆn pre-processes the sampled data directly on the satellite and flags changed areas to prioritise for downlink, shortening the response time. We verified the efficacy of our system on a dataset—which we release alongside this publication—composed of time series containing a catastrophic event, demonstrating that RaVÆn outperforms pixel-wise baselines. Finally, we tested our approach on resource-limited hardware for assessing computational and memory limitations, simulating deployment on real hardware.

Web page

Cloud cover is a major limiting factor in exploiting time-series data acquired by optical spaceborne remote sensing sensors. Multiple methods have been developed to address the problem of cloud detection in satellite imagery and a number of cloud masking algorithms have been developed for optical sensors but very few studies have carried out quantitative intercomparison of state-of-the-art methods in this domain. This paper summarizes results of the first Cloud Masking Intercomparison eXercise (CMIX) conducted within the Committee Earth Observation Satellites (CEOS) Working Group on Calibration \& Validation (WGCV). CEOS is the forum for space agency coordination and cooperation on Earth observations, with activities organized under working groups. CMIX, as one such activity, is an international collaborative effort aimed at intercomparing cloud detection algorithms for moderate-spatial resolution (10–30 m) spaceborne optical sensors. The focus of CMIX is on open and free imagery acquired by the Landsat 8 (NASA/USGS) and Sentinel-2 (ESA) missions. Ten algorithms developed by nine teams from fourteen different organizations representing universities, research centers and industry, as well as space agencies (CNES, ESA, DLR, and NASA), are evaluated within the CMIX. Those algorithms vary in their approach and concepts utilized which were based on various spectral properties, spatial and temporal features, as well as machine learning methods. Algorithm outputs are evaluated against existing reference cloud mask datasets. Those datasets vary in sampling methods, geographical distribution, sample unit (points, polygons, full image labels), and generation approaches (experts, machine learning, sky images). Overall, the performance of algorithms varied depending on the reference dataset, which can be attributed to differences in how the reference datasets were produced. The algorithms were in good agreement for thick cloud detection, which were opaque and had lower uncertainties in their identification, in contrast to thin/semi-transparent clouds detection. Not only did CMIX allow identification of strengths and weaknesses of existing algorithms and potential areas of improvements, but also the problems associated with the existing reference datasets. The paper concludes with recommendations on generating new reference datasets, metrics, and an analysis framework to be further exploited and additional input datasets to be considered by future CMIX activities.

Sentinel-3 is an Earth observation satellite constellation launched by the European Space Agency. Each satellite carries two optical multispectral instruments: the Ocean and Land Colour Instrument (OLCI) and the Sea and Land Surface Temperature Radiometer (SLSTR). OLCI and SLSTR sensors produce images covering the visible and infrared spectrum that can be collocated in order to generate synergistic products. In Earth observation, a particular weakness of optical sensors is their high sensitivity to clouds and their shadows. An incorrect cloud and cloud shadow detection leads to mistakes in both land and ocean retrievals of biophysical parameters. In order to exploit both OLCI and SLSTR capabilities, image co-registration at ground level is needed. However, applying such collocation of the images results in cloud location mismatches due to the different viewing angles of OLCI and SLSTR, which complicates the synergistic cloud detection. This study seeks to provide a solution to correctly obtain the projected clouds based on the estimation of cloud top heights in order to better collocate clouds between sensors and detect their shadows. The study presents a forward and backward method to estimate the real nadir position of a cloud on the satellite image starting from an existing cloud mask, as well as the corresponding cloud projections on the surface depending on the solar and sensor viewing angles. The estimation of cloud top heights is based on differences in the cloud projections from SLSTR nadir and oblique views. Experimental results show that the stereo cloud matching based on maximum cross-correlation between SLSTR nadir and oblique spectra was the most robust method to match SLSTR clouds for both nadir and oblique views as compared to spectral distance and spectral angle minimization. We test the method over several images around the world, leading to higher overall accuracy (OA) as compared to Sentinel-3 official products, both in detecting SLSTR clouds and OLCI cloud shadows (SLSTR nadir OA = 93.6\%, SLSTR oblique OA = 88.7\%, OLCI cloud shadow OA = 93.9\% for the stereo matcher, against 82.2\%, 81.3\% and 90.5\%, respectively, for the official Sentinel-3 products). This study also provides a starting point in the development of a cloud screening approach for the upcoming Fluorescence Explorer (FLEX) satellite mission, expected to fly in tandem with Sentinel-3.

Code

Web page

Video

Spaceborne Earth observation is a key technology for flood response, offering valuable information to decision makers on the ground. Very large constellations of small, nano satellites— ’CubeSats’ are a promising solution to reduce revisit time in disaster areas from days to hours. However, data transmission to ground receivers is limited by constraints on power and bandwidth of CubeSats. Onboard processing offers a solution to decrease the amount of data to transmit by reducing large sensor images to smaller data products. The ESA’s recent PhiSat-1 mission aims to facilitate the demonstration of this concept, providing the hardware capability to perform onboard processing by including a power-constrained machine learning accelerator and the software to run custom applications. This work demonstrates a flood segmentation algorithm that produces flood masks to be transmitted instead of the raw images, while running efficiently on the accelerator aboard the PhiSat-1. Our models are trained on WorldFloods: a newly compiled dataset of 119 globally verified flooding events from disaster response organizations, which we make available in a common format. We test the system on independent locations, demonstrating that it produces fast and accurate segmentation masks on the hardware accelerator, acting as a proof of concept for this approach.

Code

The systematic monitoring of the Earth using optical satellites is limited by the presence of clouds. Accurately detecting these clouds is necessary to exploit satellite image archives in remote sensing applications. Despite many developments, cloud detection remains an unsolved problem with room for improvement, especially over bright surfaces and thin clouds. Recently, advances in cloud masking using deep learning have shown significant boosts in cloud detection accuracy. However, these works are validated in heterogeneous manners, and the comparison with operational threshold-based schemes is not consistent among many of them. In this work, we systematically compare deep learning models trained on Landsat-8 images on different Landsat-8 and Sentinel-2 publicly available datasets. Overall, we show that deep learning models exhibit a high detection accuracy when trained and tested on independent images from the same Landsat-8 dataset (intra-dataset validation), outperforming operational algorithms. However, the performance of deep learning models is similar to operational threshold-based ones when they are tested on different datasets of Landsat-8 images (inter-dataset validation) or datasets from a different sensor with similar radiometric characteristics such as Sentinel-2 (cross-sensor validation). The results suggest that (i) the development of cloud detection methods for new satellites can be based on deep learning models trained on data from similar sensors and (ii) there is a strong dependence of deep learning models on the dataset used for training and testing, which highlights the necessity of standardized datasets and procedures for benchmarking cloud detection models in the future.

Code

Visualization

Video

The number of Earth observation satellites carrying optical sensors with similar characteristics is constantly growing. Despite their similarities and the potential synergies among them, derived satellite products are often developed for each sensor independently. Differences in retrieved radiances lead to significant drops in accuracy, which hampers knowledge and information sharing across sensors. This is particularly harmful for machine learning algorithms, since gathering new ground-truth data to train models for each sensor is costly and requires experienced manpower. In this work, we propose a domain adaptation transformation to reduce the statistical differences between images of two satellite sensors in order to boost the performance of transfer learning models. The proposed methodology is based on the cycle consistent generative adversarial domain adaptation framework that trains the transformation model in an unpaired manner. In particular, Landsat-8 and Proba-V satellites, which present different but compatible spatio-spectral characteristics, are used to illustrate the method. The obtained transformation significantly reduces differences between the image datasets while preserving the spatial and spectral information of adapted images, which is, hence, useful for any general purpose cross-sensor application. In addition, the training of the proposed adversarial domain adaptation model can be modified to improve the performance in a specific remote sensing application, such as cloud detection, by including a dedicated term in the cost function. Results show that, when the proposed transformation is applied, cloud detection models trained in Landsat-8 data increase cloud detection accuracy in Proba-V.

Forecasting crop yields is becoming increasingly important under the current context in which food security needs to be ensured despite the challenges brought by climate change, an expanding world population accompanied by rising incomes, increasing soil erosion, and decreasing water resources. Temperature, radiation, water availability and other environmental conditions influence crop growth, development, and final grain yield in a complex nonlinear manner. Machine learning (ML) techniques, and deep learning (DL) methods in particular, can account for such nonlinear relations between yield and its covariates. However, they typically lack transparency and interpretability, since the way the predictions are derived is not directly evident. Yet, in the context of yield forecasting, understanding which are the underlying factors behind both a predicted loss or gain is of great relevance. Here, we explore how to benefit from the increased predictive performance of DL methods while maintaining the ability to interpret how the models achieve their results. To do so, we applied a deep neural network to multivariate time series of vegetation and meteorological data to estimate the wheat yield in the Indian Wheat Belt. Then, we visualized and analyzed the features and yield drivers learned by the model with the use of regression activation maps. The DL model outperformed other tested models (ridge regression and random forest) and facilitated the interpretation of variables and processes that lead to yield variability. The learned features were mostly related to the length of the growing season, and temperature and light conditions during this time. For example, our results showed that high yields in 2012 were associated with low temperatures accompanied by sunny conditions during the growing period. The proposed methodology can be used for other crops and regions in order to facilitate application of DL models in agriculture.

Code

PDF

Accurate cloud detection algorithms are mandatory to analyze the large streams of data coming from the different optical Earth observation satellites. Deep learning (DL) based cloud detection schemes provide very accurate cloud detection models. However, training these models for a given sensor requires large datasets of manually labeled samples, which are very costly or even impossible to create when the satellite has not been launched yet. In this work, we present an approach that exploits manually labeled datasets from one satellite to train deep learning models for cloud detection that can be applied (or transferred) to other satellites. We take into account the physical properties of the acquired signals and propose a simple transfer learning approach using Landsat-8 and Proba-V sensors, whose images have different but similar spatial and spectral characteristics. Three types of experiments are conducted to demonstrate that transfer learning can work in both directions: (a) from Landsat-8 to Proba-V, where we show that models trained only with Landsat-8 data produce cloud masks 5 points more accurate than the current operational Proba-V cloud masking method, (b) from Proba-V to Landsat-8, where models that use only Proba-V data for training have an accuracy similar to the operational FMask in the publicly available Biome dataset (87.79–89.77\% vs 88.48\%), and (c) jointly from Proba-V and Landsat-8 to Proba-V, where we demonstrate that using jointly both data sources the accuracy increases 1–10 points when few Proba-V labeled images are available. These results highlight that, taking advantage of existing publicly available cloud masking labeled datasets, we can create accurate deep learning based cloud detection models for new satellites, but without the burden of collecting and labeling a large dataset of images.

Satellite remote sensing has been widely used in the last decades for agricultural applications, both for assessing vegetation condition and for subsequent yield prediction. Existing remote sensing-based methods to estimate gross primary productivity (GPP), which is an important variable to indicate crop photosynthetic function and stress, typically rely on empirical or semi-empirical approaches, which tend to over-simplify photosynthetic mechanisms. In this work, we take advantage of all parallel developments in mechanistic photosynthesis modeling and satellite data availability for an advanced monitoring of crop productivity. In particular, we combine process-based modeling with the soil-canopy energy balance radiative transfer model (SCOPE) with Sentinel-2 and Landsat 8 optical remote sensing data and machine learning methods in order to estimate crop GPP. With this approach, we by-pass the need for an intermediate step to retrieve the set of vegetation biophysical parameters needed to accurately model photosynthesis, while still accounting for the complex processes of the original physically-based model. Several implementations of the machine learning models are tested and validated using simulated and flux tower-based GPP data. Our final neural network model is able to estimate GPP at the tested flux tower sites with r2 of 0.92 and RMSE of 1.38 gC d−1 m−2, which outperforms empirical models based on vegetation indices. The first test of applicability of this model to Landsat 8 data showed good results (r2 of 0.82 and RMSE of 1.97 gC d−1 m−2), which suggests that our approach can be further applied to other sensors. Modeling and testing is restricted to C3 crops in this study, but can be extended to C4 crops by producing a new training dataset with SCOPE that accounts for the different photosynthetic pathways. Our model successfully estimates GPP across a variety of C3 crop types and environmental conditions even though it does not use any local information from the corresponding sites. This highlights its potential to map crop productivity from new satellite sensors at a global scale with the help of current Earth observation cloud computing platforms.

Code

Visualization

The exploitation of Earth observation satellite images acquired by optical instruments requires an automatic and accurate cloud detection. Multitemporal approaches to cloud detection are usually more powerful than their single scene counterparts since the presence of clouds varies greatly from one acquisition to another whereas surface can be assumed stationary in a broad sense. However, two practical limitations usually hamper their operational use: the access to the complete satellite image archive and the required computational power. This work presents a cloud detection and removal methodology implemented in the Google Earth Engine (GEE) cloud computing platform in order to meet these requirements. The proposed methodology is tested for the Landsat-8 mission over a large collection of manually labeled cloud masks from the Biome dataset. The quantitative results show state-of-the-art performance compared with mono-temporal standard approaches, such as FMask and ACCA algorithms, yielding improvements between 4\–5\% in classification accuracy and 3\–10\% in commission errors. The algorithm implementation within the Google Earth Engine and the generated cloud masks for all test images are released for interested readers.

Code

The colored dissolved organic matter (CDOM) variable is the standard measure of humic substance in waters optics. CDOM is optically characterized by its spectral absorption coefficient, a C D O M at at reference wavelength (e.g., \≈ 440 nm). Retrieval of CDOM is traditionally done using bio-optical models. As an alternative, this paper presents a comparison of five machine learning methods applied to Sentinel-2 and Sentinel-3 simulated reflectance ( R r s ) data for the retrieval of CDOM: regularized linear regression (RLR), random forest regression (RFR), kernel ridge regression (KRR), Gaussian process regression (GPR) and support vector machines (SVR). Two different datasets of radiative transfer simulations are used for the development and training of the machine learning regression approaches. Statistics comparison with well-established polynomial regression algorithms shows optimistic results for all models and band combinations, highlighting the good performance of the methods, especially the GPR approach, when all bands are used as input. Application to an atmospheric corrected OLCI image using the reflectance derived form the alternative neural network (Case 2 Regional) is also shown. Python scripts and notebooks are provided to interested users.

HyperLabelMe is a web platform that allows the automatic benchmarking of remote-sensing image classifiers. To demonstrate this platform's attributes, we collected and harmonized a large data set of labeled multispectral and hyperspectral images with different numbers of classes, dimensionality, noise sources, and levels. The registered user can download training data pairs (spectra and land cover/use labels) and submit the predictions for unseen testing spectra. The system then evaluates the accuracy and robustness of the classifier, and it reports different scores as well as a ranked list of the best methods and users. The system is modular, scalable, and ever-growing in data sets and classifier results.

Code

Visualization

PDF

Automatic cloud masking of Earth observation images is one of the first required steps in optical remote sensing data processing since the operational use and product generation from satellite image time series might be hampered by undetected clouds. The high temporal revisit of current and forthcoming missions and the scarcity of labeled data force us to cast cloud screening as an unsupervised change detection problem in the temporal domain. We introduce a cloud screening method based on detecting abrupt changes along the time dimension. The main assumption is that image time series follow smooth variations over land (background) and abrupt changes will be mainly due to the presence of clouds. The method estimates the background surface changes using the information in the time series. In particular, we propose linear and nonlinear least squares regression algorithms that minimize both the prediction and the estimation error simultaneously. Then, significant differences in the image of interest with respect to the estimated background are identified as clouds. The use of kernel methods allows the generalization of the algorithm to account for higher-order (nonlinear) feature relations. After the proposed cloud masking and cloud removal, cloud-free time series at high spatial resolution can be used to obtain a better monitoring of land cover dynamics and to generate more elaborated products. The method is tested in a dataset with 5-day revisit time series from SPOT-4 at high resolution and with Landsat-8 time series. Experimental results show that the proposed method yields more accurate cloud masks when confronted with state-of-the-art approaches typically used in operational settings. In addition, the algorithm has been implemented in the Google Earth Engine platform, which allows us to access the full Landsat-8 catalog and work in a parallel distributed platform to extend its applicability to a global planetary scale.

Web page

UNEP's Methane Alert and Response System (MARS) is a satellite-based system for the detection and mitigation of methane emissions around the world. As part of the International Methane Emissions Observatory (IMEO), MARS is the first global system connecting satellite methane detection to transparent notification processes intended to trigger mitigation efforts. MARS harnesses state-of-the-art satellite data to identify major emissions, activate its partners to notify relevant stakeholders, and support and track progress toward mitigation. During the year-long pilot phase, more than 600 plumes from the energy sector were detected with high-resolution satellites, and more than a hundred were notified. In December 2023, MARS entered the nominal phase with the launch of a data portal including information about the plumes detected and notified by MARS. In its current form, MARS is focused on the detection of strong point sources ({\textasciitilde}{\textgreater}1 ton/h) from the oil and gas production sector, but the system is expected to develop and integrate observations from new satellites as they become available and extend to the notification of smaller sources, also from other sectors such as coal mining, waste, or agriculture. In this contribution, we will provide a brief overview of the MARS satellite-based plume detection and monitoring system, with the updates made since the launch of the nominal phase. Furthermore, we will describe some examples of real source detection and notification efforts and discuss the next steps planned for MARS in 2024.

Code

In Earth observation, deep learning models rely heavily on comprehensive datasets for training and evaluation. However, the relevance of data quality is often underestimated, leading to subpar generalization in real-world remote sensing scenarios. This study aims to bridge this gap by proposing a straightforward method to identify critical human annotation errors in semantic segmentation datasets. The approach is based on two indices: trustworthiness and hardness. By implementing these indices, we estimate the extent of human annotation errors in CloudSEN12, a global dataset specifically designed for cloud detection in Sentinel-2 imagery. Considering only the trustworthiness index, our approach identified 1794 potential labelling errors among 10,000 image patches. Out of these, 106 were confirmed as human errors, resulting in a true positive rate of 9.86\%. When this method was applied to other extensive cloud masking datasets, such as KappaSet and Sentinel-2 Cloud Mask Catalogue, it was found that over 44\% of the human labels were inaccurate. These results do not imply the inferior quality of these datasets, instead, they highlight the considerable shift between the annotation protocols, making inter-dataset benchmarking exercises inequitable.

Code

This paper introduces DTACSNet, a Convolutional Neural Network (CNN) model specifically developed for efficient onboard atmospheric correction and cloud detection in optical Earth observation satellites. The model is developed with Sentinel-2 data. Through a comparative analysis with the operational Sen2Cor processor, DTACSNet demonstrates a significantly better performance in cloud scene classification (F2 score of 0.89 for DTACSNet compared to 0.51 for Sen2Cor v2.8) and a surface reflectance estimation with average absolute error below 2\% in reflectance units. Moreover, we tested DTACSNet on hardware-constrained systems similar to recent deployed missions and show that DTACSNet is 11 times faster than Sen2Cor with a significantly lower memory consumption footprint. These preliminary results highlight the potential of DTACSNet to provide enhanced efficiency, autonomy, and responsiveness in onboard data processing for Earth observation satellite missions.

Code

Artificial intelligence onboard satellites has the potential to reduce data transmission requirements, enable real-time decision-making and collaboration within constellations. This study deploys a lightweight foundational model called RaVAEn on D-Orbit’s ION SCV004 satellite. RaVAEn is a variational auto-encoder (VAE) that generates compressed latent vectors from small image tiles, enabling several downstream tasks. In this work we demonstrate the reliable use of RaVAEn onboard a satellite, achieving an encoding time of 0.110s for tiles of a 4.8x4.8 km2 area. In addition, we showcase fast few-shot training onboard a satellite using the latent representation of data. We compare the deployment of the model on the on-board CPU and on the available Myriad vision processing unit (VPU) accelerator. To our knowledge, this work shows for the first time the deployment of a multitask model onboard a CubeSat and the onboard training of a machine learning model.

Slides

Visualization

Video

Where are the Earth's streams flowing right now? Inland surface waters expand with floods and contract with droughts, so there is no one map of our streams. Current satellite approaches are limited to monthly observations that map only the widest streams. These are fed by smaller tributaries that make up much of the dendritic surface network but whose flow is unobserved. A complete map of our daily waters can give us an early warning for where droughts are born: the receding tips of the flowing network. Mapping them over years can give us a map of impermanence of our waters, showing where to expect water, and where not to. To that end, we feed the latest high-res sensor data to multiple deep learning models in order to map these flowing networks every day, stacking the times series maps over many years. Specifically, i) we enhance water segmentation to \$50\$ cm/pixel resolution, a 60\${\textbackslash}times\$ improvement over previous state-of-the-art results. Our U-Net trained on 30-40cm WorldView3 images can detect streams as narrow as 1-3m (30-60\${\textbackslash}times\$ over SOTA). Our multi-sensor, multi-res variant, WasserNetz, fuses a multi-day window of 3m PlanetScope imagery with 1m LiDAR data, to detect streams 5-7m wide. Both U-Nets produce a water probability map at the pixel-level. ii) We integrate this water map over a DEM-derived synthetic valley network map to produce a snapshot of flow at the stream level. iii) We apply this pipeline, which we call Pix2Streams, to a 2-year daily PlanetScope time-series of three watersheds in the US to produce the first high-fidelity dynamic map of stream flow frequency. The end result is a new map that, if applied at the national scale, could fundamentally improve how we manage our water resources around the world.

Slides

Poster

Code

Satellite imaging is a critical technology for monitoring and responding to natural disasters such as flooding. Despite the capabilities of modern satellites, there is still much to be desired from the perspective of first response organisations like UNICEF. Two main challenges are rapid access to data, and the ability to automatically identify flooded regions in images. We describe a prototypical flood segmentation system, identifying cloud, water and land, that could be deployed on a constellation of small satellites, performing processing on board to reduce downlink bandwidth by 2 orders of magnitude. We target PhiSat-1, part of the FSSCAT mission, which is planned to be launched by the European Space Agency (ESA) near the start of 2020 as a proof of concept for this new technology.

Slides

PDF

In this work, we propose to exploit both the temporal and spatial correlations in Earth observation satellite images through deep learning methods. In particular, the combination of a U-Net convolutional neural network together with a convolutional long short-term memory (LSTM) layer is proposed. This model is applied for cloud detection on MSG/SEVIRI image time series over selected landmarks. Implementation details are provided and our proposal is compared against a standard SVM and a U-Net without the convolutional LSTM layer but including temporal information too. Experimental results show that this combination of networks exploits both the spatial and temporal dependence and provides state-of-the-art classification results on this dataset.

Slides

Training machine learning algorithms for new satellites requires collecting new data. This is a critical drawback for most remote sensing applications and specially for cloud detection. A sensible strategy to mitigate this problem is to exploit available data from a similar sensor, which involves transforming this data to resemble the new sensor data. However, even taking into account the technical characteristics of both sensors to transform the images, statistical differences between data distributions still remain. This results in a poor performance of the methods trained on one sensor and applied to the new one. In this this work, we propose to use the generative adversarial networks (GANs) framework to adapt the data from the new satellite. In particular, we use Landsat-8 images, with the corresponding ground truth, to perform cloud detection in Proba-V. Results show that the GANs adaptation significantly improves the detection accuracy.

Code

Water quality parameters are derived applying several machine learning regression methods on the Case2eXtreme dataset (C2X). The used data are based on Hydrolight in-water radiative transfer simulations at Sentinel-3 OLCI wavebands, and the application is done exclusively for absorbing waters with high concentrations of coloured dissolved organic matter (CDOM). The regression approaches are: regularized linear, random forest, Kernel ridge, Gaussian process and support vector regressors. The validation is made with and an independent simulation dataset. A comparison with the OLCI Neural Network Swarm (ONSS) is made as well. The best approached is applied to a sample scene and compared with the standard OLCI product delivered by EUMETSAT/ESA.

Slides

PDF

Code

Kernel methods have been very successful in remote sensing problems because of their ability to deal with high dimensional non-linear data. However, they are computationally expensive to train when a large amount of samples are used. In this context, while the amount of available remote sensing data has constantly increased, the size of training sets in kernel methods is usually restricted to few thousand samples. In this work, we modified the kernel ridge regression (KRR) training procedure to deal with large scale datasets. In addition, the basis functions in the reproducing kernel Hilbert space are defined as parameters to be also optimized during the training process. This extends the number of free parameters from two (in the standard KRR with an RBF kernel) to more than fifty thousand in our experiments. The effectiveness of the proposal is illustrated in the problem of surface temperature estimation from MetOp-IASI hyperspectral infrared sounding data. The data set used contains more than one million samples, but the proposed method could potentially be trained with much more data.

Slides

Poster

PDF

Cloud detection is a key issue for exploiting the information from Earth observation satellites multispectral sensors. For Proba-V, cloud detection is challenging due to the limited number of spectral bands. Advanced machine learning methods, such as convolutional neural networks (CNN), have shown to work well on this problem provided enough labeled data. However, simultaneous collocated information about the presence of clouds is usually not available or requires a great amount of manual labor. In this work, we propose to learn from the available Landsat −8 cloud masks datasets and transfer this learning to solve the Proba-V cloud detection problem. CNN are trained with Landsat images adapted to resemble Proba-V characteristics and tested on a large set of real Proba-V scenes. Developed models outperform current operational Proba-V cloud detection without being trained with any real Proba-V data. Moreover, cloud detection accuracy can be further increased if the CNN are fine-tuned using a limited amount of Proba-V data.

New social and economic activities massively exploit big data and machine learning algorithms to do inference on people’s lives. Applications include automatic curricula evaluation, wage determination, and risk assessment for credits and loans. Recently, many governments and institutions have raised concerns about the lack of fairness, equity and ethics in machine learning to treat these problems. It has been shown that not including sensitive features that bias fairness, such as gender or race, is not enough to mitigate the discrimination when other related features are included. Instead, including fairness in the objective function has been shown to be more efficient.We present novel fair regression and dimensionality reduction methods built on a previously proposed fair classification framework. Both methods rely on using the Hilbert Schmidt independence criterion as the fairness term. Unlike previous approaches, this allows us to simplify the problem and to use multiple sensitive variables simultaneously. Replacing the linear formulation by kernel functions allows the methods to deal with nonlinear problems. For both linear and nonlinear formulations the solution reduces to solving simple matrix inversions or generalized eigenvalue problems. This simplifies the evaluation of the solutions for different trade-off values between the predictive error and fairness terms. We illustrate the usefulness of the proposed methods in toy examples, and evaluate their performance on real world datasets to predict income using gender and/or race discrimination as sensitive variables, and contraceptive method prediction under demographic and socio-economic sensitive descriptors.

Poster

PDF

Accurate and automatic detection of clouds in optical Earth observation satellite scenes is a key issue for a wide range of remote sensing applications. With no accurate cloud masking, undetected clouds are one of the most significant sources of error in both sea and land biophysical parameter retrieval. We present recent advances in statistical cloud detection for Proba-V imagery. The objective of the cloud masking algorithms is to detect clouds accurately providing a cloud flag per pixel. We first approach the problem from the classical machine learning perspective based on feature extraction plus supervised classification with neural networks (NN). Using this approach we significantly improve the cloud detection accuracy compared to the operational Proba-V algorithm. Then we approach the problem using deep learning methods based on convolutional neural networks (CNN). Experimental results show that CNN are a promising alternative for solving cloud masking problems.

Slides

Convolutional neural networks (CNN) have proven to be state of the art methods for many image classification tasks and their use is rapidly increasing in remote sensing problems. One of their major strengths is that, when enough data is available, CNN perform an end-to-end learning without the need of custom feature extraction methods. In this work, we study the use of different CNN architectures for cloud masking of Proba-V multispectral images. We compare such methods with the more classical machine learning approach based on feature extraction plus supervised classification. Experimental results suggest that CNN are a promising alternative for solving cloud masking problems.

Poster

Code

The vast amount of data acquired by current high resolution Earth observation satellites implies some technical challenges to be faced. Google Earth Engine (GEE) platform provides a framework for the development of algorithms and products built over this data in an easy and scalable manner. In this paper, we take advantage of the GEE platform capabilities to exploit the wealth of information in the temporal dimension by processing a long time series of satellite images. A cloud detection algorithm for Landsat-8, which uses previous images of the same location to detect clouds, is implemented and tested on the GEE platform.

This paper presents the development and implementation of a cloud detection algorithm for Proba-V. Accurate and automatic detection of clouds in satellite scenes is a key issue for a wide range of remote sensing applications. With no accurate cloud masking, undetected clouds are one of the most significant sources of error in both sea and land cover biophysical parameter retrieval. The objective of the algorithms presented in this paper is to detect clouds accurately providing a cloud flag per pixel. For this purpose, the method exploits the information of Proba-V using statistical machine learning techniques to identify the clouds present in Proba-V products. The effectiveness of the proposed method is successfully illustrated using a large number of real Proba-V images.

This paper discusses results from 12 months of a Round Robin exercise aimed at the inter-comparison of different cloud detection algorithms for Proba-V. Clouds detection is a critical issue for satellite optical remote sensing, since potential errors in cloud masking directly translates into significant uncertainty in the retrieved downstream geophysical products. Cloud detection is particularly challenging for Proba-V due to the presence of a limited number of spectral bands and the lack of thermal infrared bands. The main objective of the project was the inter-comparison of several cloud detection algorithms for Proba-V over a wide range of surface types and environmental conditions. Proba-V Level 2a products have been distributed to six different algorithm providers representing companies and research institutes in several European countries. The considered cloud detection approaches are based on different strategies: Neural Network, Discriminant Analysis, Multi-spectral and Multi-textural Thresholding, Self-Organizing Feature Maps, Dynamic Thresholding, and physically-based retrieval of Cloud Optical Thickness. The results from all algorithms were analysed and compared against a reference dataset, consisting of a large number (more than fifty thousands) of visually classified pixels. The quality assessment was performed according to a uniform methodology and the results provide clear indication on the potential best-suited approach for next Proba-V cloud detection algorithm.

In recent years, deep learning has emerged as the dominant approach for flood mapping from remote sensing satellite imagery. While new flood segmentation models are increasingly being proposed, most of these works focus on advancing architectures trained on single datasets. Therefore, these studies overlook the intrinsic limitations and biases of the available training and evaluation data. This often leads to poor generalization and overconfident predictions when these models are used in real-world scenarios. To address this gap, the objective of this work is twofold. First, we train and evaluate flood segmentation models on five publicly available datasets including data from Sentinel-1, Sentinel-2, and both SAR and multispectral modalities. Our findings reveal that models achieving high detection accuracy on a single dataset (intra-dataset validation) do not necessarily generalize well to unseen datasets. In contrast, models trained on more diverse samples from multiple datasets demonstrate greater robustness and generalization ability. Furthermore, we present a dual-stream multimodal architecture that can be independently trained and tested on both single-modality and dual-modality datasets. This enables the integration of all the diversity and richness of available data into a single unified framework. The results emphasize the need for a more comprehensive validation using diverse and well-designed datasets, particularly for multimodal approaches. If not adequately addressed, the shortcomings of current datasets can significantly limit the potential of deep learning-based flood mapping approaches.

Mitigating methane emissions is the fastest way to stop global warming in the short-term and buy humanity time to decarbonise. Despite the demonstrated ability of remote sensing instruments to detect methane plumes, no system has been available to routinely monitor and act on these events. We present MARS-S2L, an automated AI-driven methane emitter monitoring system for Sentinel-2 and Landsat satellite imagery deployed operationally at the United Nations Environment Programme's International Methane Emissions Observatory. We compile a global dataset of thousands of super-emission events for training and evaluation, demonstrating that MARS-S2L can skillfully monitor emissions in a diverse range of regions globally, providing a 216\% improvement in mean average precision over a current state-of-the-art detection method. Running this system operationally for six months has yielded 457 near-real-time detections in 22 different countries of which 62 have already been used to provide formal notifications to governments and stakeholders.

Earth observation through remote sensing sensors in orbiting satellites provide us with a great capacity to monitor our planet at high spatial and temporal resolutions. Nevertheless, to process all this ever-growing amount of data, we need to develop fast and accurate models adapted to the specific characteristics of the data of each sensor. For optical sensors, detecting the clouds in the image is an unavoidable first step to most of the land and ocean applications. Although detecting bright and opaque clouds is relatively easy, automatically identifying thin semi-transparent clouds or differentiating clouds from snow or bright surfaces is much more challenging. In addition, in the current scenario where the number of sensors in space is constantly growing, developing methodologies to transfer models across different satellite data is a pressing need. Henceforth, the overreaching goal of this Thesis is to develop accurate cloud detection models that exploit the different properties of the satellite images and to develop methodologies to transfer those models across different sensors. The four contributions of this Thesis are stepping stones in that direction. In the first contribution, "Multitemporal cloud masking in the Google Earth Engine", we implemented a lightweight multitemporal cloud detection model that runs on the Google Earth Engine platform and which outperforms the operational models for Landsat-8. The second contribution, "Transferring deep learning models for Cloud Detection between Landsat-8 and Proba-V", is a case-study of transferring a deep learning based cloud detection algorithm from Landsat-8 (30m resolution, 12 spectral bands and very good radiometric quality) to Proba-V which has a lower 333m resolution, only four bands and less good radiometric quality. The third paper "Cross sensor adversarial domain adaptation of Landsat-8 and Proba-V images for cloud detection" propose a learning-based domain adaptation transformation to Proba-V images to resemble those taken by Landsat-8 with the objective of transferring products designed on Landsat-8 to Proba-V. Finally, the fourth contribution, "Towards global flood mapping onboard low cost satellites with machine learning", tackles simultaneously cloud and flood water detection with a single deep learning model; in this case the model is implemented so that it could run onboard of a CubeSat (Phi-Sat) with an AI accelerator chip; the model is trained on Sentinel-2 images and we demonstrate how to transfer this model to the Phi-Sat camera. We trained this model in a newly compiled dataset of more than 100 verified flood events called WorldFloods. This model was launched on June 2021 onboard the Wild Ride D-Orbit mission and we are testing now its performance in space.

One of the most exciting trends in machine learning nowadays is the use of deep networks to learn density properties. In this direction, the field of generative models has become a hot topic in deep learning, and different approaches have been proposed. In this chapter, we are going to overview some of the generative models based on deep learning and focus on one of the most used ones: the Generative Adversarial Networks (GANs). We review the main different families of GANs, and how these families have been applied to remote sensing problems. Finally, in the last section, we illustrate the use of GANs in remote sensing problems with two different applications.

Earth observation from satellites poses challenging problems where machine learning is being widely adopted as a key player. Perhaps the most challenging scenario that we are facing nowadays is to provide accurate estimates of particular variables of interest characterizing the Earth's surface. This chapter introduces some recent advances in statistical bio-geophysical parameter retrieval from satellite data. In particular, we will focus on Gaussian process regression (GPR) that has excelled in parameter estimation as well as in modeling complex radiative transfer processes. GPR is based on solid Bayesian statistics and generally yields efficient and accurate parameter estimates, accompanied by confidence intervals for the estimations. We will first review the standard application of GPs for parameter retrieval and inverse modeling, which is based on regressing the parameter of interest on the concurrent observations. Secondly, noting that very often several parameters need to be estimated simultaneously, we review the field of multioutput GPR and illustrate its application in gap filling of multiple time series of parameters. The third GP modeling that we review here is that of combining real and simulated data. Very often a forward model encoding the well-understood physical relations is available. Inverting the model with GP is a standard practice known as hybrid modeling. In addition, we review a joint GP (JGP) model that combines both in situ measurements and simulated data in a single GP model and allows us to transfer information across spatial, temporal, and land cover modalities such as different crops. In recent years, forced by the data deluge, a plethora of large-scale GP models were introduced. We review recent advances and illustrate their performance for the estimation of surface temperature from infrared sounding data. Finally, we take a reversed pathway and focus on mimicking physical models with GPs. We present an automatic emulation scheme that approximates the forward physical model via interpolation, reducing the number of necessary nodes. Empirical evidence of the performance of these models will be presented through illustrative examples of land and vegetation monitoring.

The motivation of this work was to be introduced into the mathematical theory of Stochastic Processes with the purpose of getting a solid background. The following list remarks some of the most important achievements of the project: 1. The work constitutes an introduction to Stochastic Calculus and Continuous time stochastic processes. We went rigorously through the theory of Brownian motion proving some of its most important theorems. In addition, we have developed with high precision the theory of Itô integral pointing out their possible generalizations when we change the integrand process. 2. We presented, as a motivation for the theory, some of the most important applications of stochastic Calculus to Finance. 3. In the last chapter, fractional Brownian motion has been introduced. As it is explained, this process is gaining importance to model situations where the assumptions of Brownian motion are unrealistic. 4. We went through some of the most famous books in stochastic calculus, Brownian motion and martingale theory highlighting several passages of those books that contains the proofs of the theorems or a further development of the exposed ideas. This knowledge will be very useful since this work could serve as a guide through some of this texts.