Modal

Modal is a community skill for running serverless compute workloads on the Modal platform, covering function deployment, GPU provisioning, container configuration, scheduled jobs, and volume management for cloud compute workflows.

What Is This?

Overview

Modal provides tools for deploying and running compute workloads on Modal's serverless cloud infrastructure. It covers function deployment that wraps Python functions as remote endpoints that execute on cloud containers with automatic scaling, GPU provisioning that attaches specified GPU types to function containers for ML inference and training workloads, container configuration that defines Python environments with pip dependencies and system packages for reproducible execution, scheduled jobs that run functions on cron schedules for recurring batch processing tasks, and volume management that provides persistent storage across function invocations for caching models and data. The skill enables developers to run Python workloads on cloud infrastructure without managing servers or configuring underlying compute resources manually.

Who Should Use This

This skill serves ML engineers running inference workloads on cloud GPUs, backend developers deploying serverless Python functions, and data engineers scheduling batch processing jobs that require scalable compute on demand.

Why Use It?

Problems It Solves

Running GPU workloads locally requires expensive hardware purchases and ongoing maintenance. Container deployment on traditional cloud platforms involves Dockerfile writing, image building, and orchestration configuration. Serverless functions on standard cloud platforms lack GPU support needed for ML inference workloads. Persistent data between serverless invocations requires external storage configuration.

Core Highlights

Function deployer wraps Python code as cloud endpoints with automatic container management. GPU attacher provisions specified accelerator types such as T4, A10G, or A100 for compute-intensive functions. Environment builder defines reproducible container images from dependency lists. Volume manager provides persistent cross-invocation storage.

How to Use It?

Basic Usage

import modal

app = modal.App(
  'my-app')

image = modal.Image\
  .debian_slim()\
  .pip_install(
    'torch',
    'transformers')

@app.function(
  image=image,
  gpu='T4'
)
def predict(
  text: str
) -> str:
  from transformers\
    import pipeline
  pipe = pipeline(
    'sentiment-analysis')
  result = pipe(text)
  return str(result)

@app.function(
  image=image,
  schedule=modal.Cron(
    '0 */6 * * *')
)
def batch_job():
  items = fetch_items()
  for item in items:
    process(item)

Real-World Examples

import modal

app = modal.App(
  'model-serve')
volume = modal.Volume\
  .from_name(
    'model-cache',
    create_if_missing=(
      True))

image = modal.Image\
  .debian_slim()\
  .pip_install(
    'torch',
    'transformers')

@app.cls(
  image=image,
  gpu='A10G',
  volumes={
    '/cache': volume}
)
class ModelServer:
  @modal.enter()
  def load(self):
    from transformers\
      import (
        AutoModelFor\
          CausalLM,
        AutoTokenizer)
    name = (
      'meta-llama/'
      'Llama-3.2-1B')
    self.tokenizer = (
      AutoTokenizer
        .from_pretrained(
          name,
          cache_dir=(
            '/cache')))
    self.model = (
      AutoModelFor\
        CausalLM
        .from_pretrained(
          name,
          cache_dir=(
            '/cache')))

  @modal.method()
  def generate(
    self,
    prompt: str
  ) -> str:
    inputs = (
      self.tokenizer(
        prompt,
        return_tensors=(
          'pt')))
    out = self.model\
      .generate(
        **inputs,
        max_new_tokens=(
          128))
    return self.tokenizer\
      .decode(
        out[0],
        skip_special_tokens
          =True)

Advanced Tips

Use Modal volumes to cache downloaded model weights so they persist across container restarts, avoiding repeated downloads on every cold start. Deploy class-based endpoints with the enter decorator to load models once at container startup rather than per request, significantly reducing per-request latency. Use concurrent container limits to control costs during high-traffic inference serving.

When to Use It?

Use Cases

Deploy an LLM inference endpoint on GPU instances that scale to zero when idle. Run a scheduled batch processing job on cloud compute without managing server infrastructure. Serve a computer vision model with A10G GPUs and persistent model weight caching. Parallelize large dataset processing by mapping functions across many containers simultaneously.

Related Topics

Serverless compute, Modal, GPU cloud, container deployment, ML inference, batch processing, and cloud functions.

Important Notes

Requirements

Modal account with API token configured. Modal Python package installed locally. Internet connectivity for function deployment and remote invocation.

Usage Recommendations

Do: use container image caching by defining stable dependency layers to speed up cold starts. Attach volumes for large model weights to avoid downloading on every container start. Monitor function invocation counts and GPU time for cost management.

Don't: include large files in the function code that gets serialized to the cloud. Run long-duration GPU functions without setting timeout limits to prevent unexpected costs. Deploy functions without testing locally first using Modal's local execution mode.

Limitations

Cold start latency occurs when new containers need to be provisioned and initialized for the first incoming request. GPU availability depends on Modal's infrastructure capacity and may have queue times during periods of high demand. Function execution has configurable timeout limits that constrain very long running workloads.