Generate text with Flash and transformers

Flash is now part of the Runpod SDK. This page documents the standalone runpod-flash package, which stops getting fixes and updates after its deprecation date. For new projects, use the Runpod SDK. To see how Flash commands and imports map to the SDK, see Migrating from Flash.

This tutorial shows you how to build a text generation script using Flash and Hugging Face’s transformers library. You’ll learn how to load a pretrained language model on a GPU worker and generate text from prompts.

Requirements

You’ve created a Runpod account.
You’ve created a Runpod API key.
You’ve installed Python 3.10 to 3.14.
You’ve completed the Flash quickstart or are familiar with Flash basics.

What you’ll build

By the end of this tutorial, you’ll have a working text generation application that:

Accepts text prompts as input.
Generates natural language completions using GPT-2.
Runs entirely on Runpod’s GPU infrastructure.
Returns generated text with execution metadata.

Step 1: Set up your project

Create a new directory for your project and set up a Python virtual environment:

mkdir flash-text-generation
cd flash-text-generation

Install Flash using uv:

uv venv
source .venv/bin/activate
uv pip install runpod-flash

flash login

Alternatively, create a .env file with your API key. Flash loads .env from the current directory automatically:

echo "RUNPOD_API_KEY=YOUR_API_KEY" > .env

Replace YOUR_API_KEY with your API key from the Runpod console.

The directory name becomes your Flash app name. Run flash deploy and your script from this directory.

Step 2: Understand the Hugging Face transformers library

Hugging Face transformers is a Python library for running pretrained language models. The same API loads thousands of models from the Hugging Face Hub. This tutorial uses GPT-2, a 124M-parameter model from OpenAI. It’s small (about 500 MB), loads in seconds, and needs less than 2 GB of VRAM. It continues text rather than following instructions, so expect creative completions, not answers.

Step 3: Create your project file

Create a file called text_generation.py in your project directory. The next three steps build it up section by section. The complete file is at the end of Step 6.

Step 4: Add imports and configuration

import asyncio

from runpod_flash import Endpoint, GpuGroup

MODEL_NAME = "gpt2"

# Each worker keeps the loaded model here between requests.
_model = None
_tokenizer = None

_model and _tokenizer cache the model on the worker, so only the first request on each worker loads it.

Step 5: Define the text generation function

Add the endpoint function that runs on the GPU worker:

@Endpoint(
    name="text-generation",
    gpu=[GpuGroup.AMPERE_24, GpuGroup.ADA_24],  # 24 GB GPUs
    workers=3,
    idle_timeout=600,  # Keep workers warm for 10 minutes
    dependencies=["transformers", "torch"],
)
def generate_text(prompt, max_new_tokens=50, temperature=0.7):
    """Continue a prompt with GPT-2."""
    import time

    import torch
    from transformers import AutoModelForCausalLM, AutoTokenizer

    global _model, _tokenizer
    cached = _model is not None
    start = time.perf_counter()
    if _model is None:
        _tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
        _model = AutoModelForCausalLM.from_pretrained(MODEL_NAME).to("cuda")
    load_seconds = time.perf_counter() - start

    inputs = _tokenizer(prompt, return_tensors="pt").to("cuda")
    start = time.perf_counter()
    with torch.no_grad():
        outputs = _model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=True,
            temperature=temperature,
            pad_token_id=_tokenizer.eos_token_id,
        )
    generate_seconds = time.perf_counter() - start

    return {
        "generated_text": _tokenizer.decode(outputs[0], skip_special_tokens=True),
        "gpu": torch.cuda.get_device_name(0),
        "model_cached": cached,
        "load_seconds": round(load_seconds, 1),
        "generate_seconds": round(generate_seconds, 2),
    }

Configuration breakdown:

name="text-generation": Identifies your endpoint in the Runpod console.
gpu=[GpuGroup.AMPERE_24, GpuGroup.ADA_24]: Uses 24 GB GPUs such as the RTX 3090, RTX 4090, L4, or A5000. GPT-2 needs far less, so any of them work.
workers=3: Allows up to 3 parallel workers for concurrent requests.
idle_timeout=600: Keeps a worker running for 10 minutes after its last request, so the loaded model stays in memory.
dependencies: Installed into your build. torch is already in the worker image, so Flash leaves it out of the upload.

Don’t add accelerate to dependencies. GPT-2 doesn’t need it, and because it requires torch, on Linux flash deploy downloads torch and several GB of CUDA libraries into the local build. Without it, the build is about 40 MB.

Code breakdown

Imports inside the function: torch and transformers are imported inside generate_text because they’re installed on the worker, not on your machine.Model caching:

global _model, _tokenizer
if _model is None:
    _tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
    _model = AutoModelForCausalLM.from_pretrained(MODEL_NAME).to("cuda")

The worker imports your file once and keeps it in memory between requests, so module-level variables persist. The first request on a worker downloads and loads GPT-2. Later requests on the same worker reuse it. model_cached in the result tells you which happened.Tokenization:

inputs = _tokenizer(prompt, return_tensors="pt").to("cuda")

Converts the prompt into token IDs on the GPU.Generation parameters:

max_new_tokens: How many tokens to generate after the prompt.
do_sample=True: Samples from the model’s predictions instead of always picking the most likely token, so each run gives a different completion.
temperature: Controls randomness when sampling. Lower values are more predictable, higher values more varied.
pad_token_id: GPT-2 has no padding token, so this reuses the end-of-text token and avoids a warning.

No gradient tracking: torch.no_grad() skips the bookkeeping needed for training, which saves memory during inference.

Step 6: Add the main function

Add a main() function that calls the endpoint. It runs on your machine:

async def main():
    print("Generating text with GPT-2 on a Runpod GPU...")
    result = await generate_text("The future of artificial intelligence is", max_new_tokens=60)
    print(f"\n{result['generated_text']}\n")
    print(f"GPU: {result['gpu']}")
    print(f"Model already loaded: {result['model_cached']}")
    print(f"Model load: {result['load_seconds']}s, generation: {result['generate_seconds']}s")


if __name__ == "__main__":
    asyncio.run(main())

Keep the if __name__ == "__main__": block at the end of the file. Without it, the script exits without doing anything.

Complete file

text_generation.py

import asyncio

from runpod_flash import Endpoint, GpuGroup

MODEL_NAME = "gpt2"

# Each worker keeps the loaded model here between requests.
_model = None
_tokenizer = None


@Endpoint(
    name="text-generation",
    gpu=[GpuGroup.AMPERE_24, GpuGroup.ADA_24],  # 24 GB GPUs
    workers=3,
    idle_timeout=600,  # Keep workers warm for 10 minutes
    dependencies=["transformers", "torch"],
)
def generate_text(prompt, max_new_tokens=50, temperature=0.7):
    """Continue a prompt with GPT-2."""
    import time

    import torch
    from transformers import AutoModelForCausalLM, AutoTokenizer

    global _model, _tokenizer
    cached = _model is not None
    start = time.perf_counter()
    if _model is None:
        _tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
        _model = AutoModelForCausalLM.from_pretrained(MODEL_NAME).to("cuda")
    load_seconds = time.perf_counter() - start

    inputs = _tokenizer(prompt, return_tensors="pt").to("cuda")
    start = time.perf_counter()
    with torch.no_grad():
        outputs = _model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=True,
            temperature=temperature,
            pad_token_id=_tokenizer.eos_token_id,
        )
    generate_seconds = time.perf_counter() - start

    return {
        "generated_text": _tokenizer.decode(outputs[0], skip_special_tokens=True),
        "gpu": torch.cuda.get_device_name(0),
        "model_cached": cached,
        "load_seconds": round(load_seconds, 1),
        "generate_seconds": round(generate_seconds, 2),
    }


async def main():
    print("Generating text with GPT-2 on a Runpod GPU...")
    result = await generate_text("The future of artificial intelligence is", max_new_tokens=60)
    print(f"\n{result['generated_text']}\n")
    print(f"GPU: {result['gpu']}")
    print(f"Model already loaded: {result['model_cached']}")
    print(f"Model load: {result['load_seconds']}s, generation: {result['generate_seconds']}s")


if __name__ == "__main__":
    asyncio.run(main())

Step 7: Deploy and run your first generation

Deploy the endpoint from your project directory:

flash deploy

# If using uv:
uv run flash deploy

✓ built flash-text-generation  1 files, 1 deps, 37.8 MB
No app 'flash-text-generation' found. Creating app and 'production' environment...

✓ uploaded  37.8 MB  2.1s
✓ deployed to production  1.9s

  text-generation  https://api.runpod.ai/v2/abc123xyz/runsync

curl -X POST https://api.runpod.ai/v2/abc123xyz/runsync \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -d '{"input": {}}'

Then run the script from the same directory:

python text_generation.py

The first run takes about a minute while a worker starts and loads the model:

Generating text with GPT-2 on a Runpod GPU...

The future of artificial intelligence is bright, and it will be exciting to see if we can add a few more layers of complexity into our lives to become a better person.

GPU: NVIDIA GeForce RTX 4090
Model already loaded: False
Model load: 9.1s, generation: 0.76s

Run it again while the worker is warm. The model is already loaded, so the whole run takes about 2 seconds:

Model already loaded: True
Model load: 0.0s, generation: 0.51s

Your completions will differ because do_sample=True picks tokens at random.

The first request on a new worker can take longer than the 90 seconds runpod-flash 1.21.0 waits for a result, especially if the worker has to wait for a free GPU. If the script fails with timed out after 90s. the endpoint may not be deployed or the worker is still starting or KeyError: 'generated_text', the endpoint is deployed and the worker is still starting. Run the script again. See First run fails after 90 seconds.

Step 8: Try several prompts in parallel

Replace main() with a version that sends three prompts at once:

async def main():
    prompts = [
        "Once upon a time in a distant galaxy",
        "The secret to happiness is",
        "In the year 2050, technology will",
    ]
    results = await asyncio.gather(
        *(generate_text(prompt, max_new_tokens=40) for prompt in prompts)
    )
    for result in results:
        print(f"\n{result['generated_text']}")


if __name__ == "__main__":
    asyncio.run(main())

You only changed code outside the @Endpoint function, so you don’t need to redeploy. Run it again:

python text_generation.py

All three requests are sent at once. A warm worker returned all three completions in about 3 seconds. If requests queue up, Flash starts more workers, up to workers=3, and each new worker loads the model on its first request.

Troubleshooting

Model download fails

Issue: Error: Failed to download model from Hugging Face. Solutions:

Check internet connectivity from workers (rare issue on Runpod).
Try a different model that might be available faster.

Increase execution timeout in configuration:

@Endpoint(
    name="text-generation",
    gpu=GpuGroup.ADA_24,
    execution_timeout_ms=300000  # 5 minutes
)

Out of memory error

Issue: RuntimeError: CUDA out of memory. Solutions:

Use smaller models (GPT-2 instead of GPT-2 Large).
Reduce max_new_tokens.
Use larger GPUs:
gpu=GpuGroup.AMPERE_48 # 48GB GPUs

Slow generation

Issue: Every request takes much longer than a few seconds. Check model_cached in the result. If it’s False on most requests, workers are loading the model again:

Make sure _model and _tokenizer are module-level variables and generate_text declares global _model, _tokenizer.

Increase idle_timeout so workers stay up between requests:

idle_timeout=1800  # Keep workers warm for 30 minutes

Set workers=(1, 3) to keep one worker running at all times. You pay for that worker while it’s idle.

If model_cached is True and generation is still slow, reduce max_new_tokens.

First run fails after 90 seconds

Issue: The first run fails with one of these errors:

RuntimeError: request to endpoint 'text-generation' timed out after 90s. the endpoint may not be deployed or the worker is still starting. deploy with 'flash deploy' or increase timeout with FLASH_SENTINEL_TIMEOUT env var.

KeyError: 'generated_text'

Cause: runpod-flash 1.21.0 waits up to 90 seconds for a result. The first request on a new worker can take longer: the worker may wait for a free GPU, then it starts and loads the model. The endpoint is deployed, and the request keeps running after the error. Solution: Run the script again. The worker keeps starting after the error, so the next run is faster. Raising FLASH_SENTINEL_TIMEOUT above 90 doesn’t help in 1.21.0: the request returns at 90 seconds with the job still running, and the script fails with the KeyError instead. To see whether a worker is up, check the endpoint’s health with its ID from flash deploy. curl reads your API key from the RUNPOD_API_KEY environment variable, which flash login doesn’t set, so export it first:

export RUNPOD_API_KEY="your_key"
curl https://api.runpod.ai/v2/ENDPOINT_ID/health \
  -H "Authorization: Bearer $RUNPOD_API_KEY"

A non-zero workers.throttled means the worker is waiting for a GPU. A non-zero workers.idle, workers.ready, or workers.running means a worker is up.

Generation quality is poor

Issue: Generated text is incoherent or repetitive. Solutions:

Adjust temperature (try 0.7-0.9)

Add top_p and top_k sampling:

outputs = _model.generate(
    **inputs,
    max_new_tokens=max_new_tokens,
    temperature=0.8,
    top_p=0.9,
    top_k=50,
    do_sample=True
)

Try a larger model (GPT-2 Medium or Large).

Next steps

Now that you’ve built a text generation script with Flash, you can:

Explore other models

Try different models from Hugging Face: Change MODEL_NAME and run flash deploy again:

# Larger general-purpose model (1.3B parameters)
MODEL_NAME = "facebook/opt-1.3b"

# Code generation model
MODEL_NAME = "Salesforce/codegen-350M-mono"

# Dialogue model
MODEL_NAME = "microsoft/DialoGPT-medium"

Build a chat interface

Extend your app to handle multi-turn conversations:

@Endpoint(
    name="chat",
    gpu=GpuGroup.ADA_24,
    dependencies=["transformers", "torch"]
)
def chat(conversation_history):
    """Multi-turn chat with context."""
    # Concatenate conversation history
    prompt = "\n".join(conversation_history)
    # Generate response
    # Return new message

Deploy as a Flash app

Convert your script to a production Flash app:

flash init text-generation-app
cd text-generation-app
# Move generate_text into gpu_worker.py
# Add HTTP routes in lb_worker.py
flash deploy

flash init creates gpu_worker.py, cpu_worker.py, and lb_worker.py at the project root.

When deploying queue-based functions with flash deploy, each function must have its own unique endpoint configuration. generate_text and chat in this tutorial already have separate configs (name="text-generation" and name="chat"), so keep them that way when you move them into a Flash app. See understanding endpoint architecture for details.

Optimize performance

Use network volumes to cache models across workers.
Implement request batching for higher throughput.
Try quantized models for faster inference.

Introduction

Serverless

Flash

Pods

Public Endpoints

Generate text with Flash and transformers

Requirements

What you’ll build

Step 1: Set up your project

Step 2: Understand the Hugging Face transformers library

Step 3: Create your project file

Step 4: Add imports and configuration

Step 5: Define the text generation function

Step 6: Add the main function

Complete file

Step 7: Deploy and run your first generation

Step 8: Try several prompts in parallel

Troubleshooting

Model download fails

Out of memory error

Slow generation

First run fails after 90 seconds

Generation quality is poor

Next steps

Explore other models

Build a chat interface

Deploy as a Flash app

Optimize performance

​Requirements

​What you’ll build

​Step 1: Set up your project

​Step 2: Understand the Hugging Face transformers library

​Step 3: Create your project file

​Step 4: Add imports and configuration

​Step 5: Define the text generation function

​Step 6: Add the main function

​Complete file

​Step 7: Deploy and run your first generation

​Step 8: Try several prompts in parallel

​Troubleshooting

​Model download fails

​Out of memory error

​Slow generation

​First run fails after 90 seconds

​Generation quality is poor

​Next steps

​Explore other models

​Build a chat interface

​Deploy as a Flash app

​Optimize performance

​Related resources

Requirements

What you’ll build

Step 1: Set up your project

Step 2: Understand the Hugging Face transformers library

Step 3: Create your project file

Step 4: Add imports and configuration

Step 5: Define the text generation function

Step 6: Add the main function

Complete file

Step 7: Deploy and run your first generation

Step 8: Try several prompts in parallel

Troubleshooting

Model download fails

Out of memory error

Slow generation

First run fails after 90 seconds

Generation quality is poor

Next steps

Explore other models

Build a chat interface

Deploy as a Flash app

Optimize performance

Related resources