Streaming satellite data to cloud workflows for on-demand computing of environmental data products
- 1 November 2010
- proceedings article
- Published by Institute of Electrical and Electronics Engineers (IEEE)
Abstract
Environmental data arriving constantly from satellites and weather stations are used to compute weather coefficients that are essential for agriculture and viticulture. For example, the reference evapotranspiration (ET0) coefficient, overlaid on regional maps, is provided each day by the California Department of Water Resources to local farmers and turf managers to plan daily water use. Scaling out single-processor compute/data intensive applications operating on realtime data to support more users and higher-resolution data poses data engineering challenges. Cloud computing helps data providers expand resource capacity to meet growing needs besides supporting scientific needs like reprocessing historic data using new models. In this article, we examine migration of a legacy script used for daily ET 0 computation by CIMIS to a workflow model that eases deployment to and scaling on the Windows Azure Cloud. Our architecture incorporates a direct streaming model into Cloud virtual machines (VMs) that improves the performance by 130% to 160% for our workflow over using Cloud storage for data staging, used commonly. The streaming workflows achieve runtimes comparable to desktop execution for single VMs and a linear speed-up when using multiple VMs, thus allowing computation of environmental coefficients at a much larger resolution than done presently.Keywords
This publication has 15 references indexed in Scilit:
- SciCumulus: A Lightweight Cloud Middleware to Explore Many Task Computing Paradigm in Scientific WorkflowsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2010
- Bridging the Gap between Desktop and the Cloud for eScience ApplicationsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2010
- Seeking supernovae in the cloudsPublished by Association for Computing Machinery (ACM) ,2010
- Early observations on the performance of Windows AzurePublished by Association for Computing Machinery (ACM) ,2010
- Scientific workflows and cloudsXRDS: Crossroads, The ACM Magazine for Students, 2010
- Building the Trident Scientific Workflow Workbench for Data Management in the CloudPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2009
- Daily reference evapotranspiration for California using satellite imagery and weather station measurement interpolationCivil Engineering and Environmental Systems, 2009
- Shedding Light on the Dark Data in the Long Tail of ScienceLibrary Trends, 2008
- Taverna: lessons in creating a workflow environment for the life sciencesConcurrency and Computation: Practice and Experience, 2005
- The evolution of environmental management: from stage models to performance evaluationBusiness Strategy and the Environment, 2001