Guidance

import guidance

@guidance
def extract_entity(
  lm,
  text: str
):
  lm += f'''Extract entity """
{text}
"""
Name: {guidance.gen(
  name="name",
  stop="\n")}
Type: {guidance.select(
  options=[
    "person",
    "organization",
    "location",
    "product"],
  name="type")}
Confidence: {guidance.gen(
  name="confidence",
  regex=(
    r"[0-9]\\.[0-9]{2}"),
  stop="\n")}
'''
  return lm

model = guidance.models\
  .LlamaCpp(
    'model.gguf')
result = model +\
  extract_entity(
    'OpenAI released '
    'GPT-4 in 2023')
print(result['name'])
print(result['type'])

import guidance

@guidance
def generate_json(
  lm,
  description: str
):
  lm += f'''Generate a JSON
object for: {description}
```json
{guidance.json(
  name="output",
  schema={{
    "type": "object",
    "properties": {{
      "title": {{
        "type": "string"}},
      "tags": {{
        "type": "array",
        "items": {{
          "type":
            "string"}}}},
      "priority": {{
        "type": "integer",
        "minimum": 1,
        "maximum": 5}}}},
    "required": [
      "title",
      "tags",
      "priority"]}})}

### Advanced Tips

Chain multiple guidance functions together to build multi-step generation pipelines where earlier outputs inform later generation prompts. Use select with computed options to create dynamic constrained choices based on runtime context. Combine regex constraints with stop tokens for fields that need both format validation and length control.

## When to Use It?

### Use Cases

Generate guaranteed valid JSON from language model outputs for API response construction. Build structured extraction pipelines that produce typed entity data from unstructured text. Create interactive generation workflows where tool results feed back into subsequent generation steps.

### Related Topics

Structured generation, constrained decoding, prompt engineering, JSON schema, grammar-guided generation, LLM applications, and output validation.

## Important Notes

### Requirements

Guidance library installed with appropriate model backend support. Compatible language model that supports token-level logit access for constrained decoding. Python 3.9 or newer for type annotation features.

### Usage Recommendations

**Do:** define explicit schemas for JSON generation to leverage grammar enforcement. Test templates with varied inputs to verify constraint satisfaction across edge cases. Use select for categorical fields and regex for formatted values.

**Don't:** apply overly restrictive constraints that prevent the model from generating coherent content within the allowed token space. Mix constrained and unconstrained generation without clear boundaries between structured and free text sections. Assume all model backends support every constraint type equally.

### Limitations

Constrained decoding requires token-level logit access which is not available through all API providers or cloud-hosted model endpoints. Complex grammars can slow generation speed since each token must be validated against the grammar constraint at decode time. Guidance templates are tightly coupled to specific model interfaces and may need updates when switching between different backends.