Coiled

Load data from cloud storage (e.g., S3) into a Dask dataframe.

Define your data processing steps using Dask operations.

Submit the computation to a Coiled cluster with thousands of workers.

Monitor the progress of the computation using the Coiled dashboard.

Save the results back to cloud storage.

Load your training data into a Dask array or dataframe.

Define your machine learning model using a framework like PyTorch or TensorFlow.

Use Dask-ML or other distributed training libraries to distribute the training process across a Coiled cluster with multiple GPUs.

Monitor the training progress using the Coiled dashboard and TensorBoard.

Save the trained model to cloud storage.

Load geospatial data from cloud storage (e.g., GeoTIFF files) into an Xarray DataArray.

Define your geospatial processing steps using Xarray operations.

Submit the computation to a Coiled cluster.

Monitor the progress of the computation using the Coiled dashboard.

Save the processed data back to cloud storage.

Define your ETL pipeline using a workflow management tool like Prefect.

Configure your Prefect tasks to run on a Coiled cluster.

Schedule your pipeline to run automatically using Prefect's scheduling features.

Monitor the execution of your pipeline using the Prefect and Coiled dashboards.

Automatically scale your Coiled cluster up or down based on the pipeline's resource requirements.

Write your batch job script, specifying hardware requirements in comments (e.g., #COILED memory 32GB).

Submit your script to Coiled using the Coiled API or CLI.

Coiled automatically provisions the necessary resources and executes your script.

Monitor the progress of your batch jobs using the Coiled dashboard.

Collect the results of your batch jobs from cloud storage.