BAD:                          GOOD:
├── Data Analyst              ├── Data Analyst
├── Business Analyst          ├── Analytics Engineer
├── Analytics (overlap!)      ├── Data Engineer
└── BI Developer (overlap!)   └── Data Scientist

BAD (org-chart thinking):     GOOD (practitioner thinking):
├── Engineering               ├── Data Engineer
│   └── Data                  ├── ML Engineer
├── Analytics                 ├── Analytics Engineer
│   └── Data                  └── Data Analyst
└── Science
    └── Data

STABLE CORE:                  EDGE HANDLING:
├── Data Engineer             "AI Engineer" → classify as:
├── ML Engineer               - ML Engineer (if model-focused)
├── Data Scientist            - out_of_scope (if API integration)
└── Analytics Engineer        Document decision, revisit quarterly

GOOD (embeddable description):
"Analytics Engineer: Builds and maintains data transformation 
pipelines using tools like dbt, creates metrics layers and 
semantic models, bridges raw data and analyst-ready datasets."

BAD (list of keywords):
"Analytics Engineer: dbt, SQL, data modeling, metrics"

job_families:
  product:
    description: "Roles focused on product strategy, discovery, and delivery"
    subfamilies:
      core_pm:
        label: "Core PM"
        description: "General product management for user-facing features"
        signals:
          titles: ["Product Manager", "PM", "Product Lead"]
          keywords: ["roadmap", "user stories", "stakeholders", "prioritization"]
          anti_signals: ["growth", "platform", "API", "ML", "AI"]
        
      growth_pm:
        label: "Growth PM"
        description: "Acquisition, retention, monetization, conversion optimization"
        signals:
          titles: ["Growth PM", "Growth Product Manager"]
          keywords: ["acquisition", "retention", "conversion", "funnel", "experimentation", "A/B testing"]
          
      platform_pm:
        label: "Platform PM"
        description: "Developer tools, APIs, infrastructure products"
        signals:
          titles: ["Platform PM", "API PM", "Developer Experience PM"]
          keywords: ["API", "SDK", "developer", "platform", "infrastructure", "internal tools"]
          
      technical_pm:
        label: "Technical PM"
        description: "Deep technical skills required, often ex-engineers"
        signals:
          titles: ["Technical Product Manager", "TPM"]
          keywords: ["technical requirements", "engineering background", "system design"]
          
      ai_ml_pm:
        label: "AI/ML PM"
        description: "AI/ML products, models, data products"
        signals:
          titles: ["AI PM", "ML PM", "AI Product Manager", "Data Product Manager"]
          keywords: ["machine learning", "AI", "model", "LLM", "GenAI", "data product"]

  data:
    description: "Roles focused on data infrastructure, analysis, and machine learning"
    subfamilies:
      product_analytics:
        label: "Product Analytics"
        description: "Product metrics, experiments, user behavior, growth analytics"
        signals:
          titles: ["Product Analyst", "Growth Analyst", "Product Data Analyst"]
          keywords: ["product metrics", "experimentation", "user behavior", "funnel analysis", "Amplitude", "Mixpanel"]
          anti_signals: ["pipeline", "infrastructure", "model training"]
          
      data_analyst:
        label: "Data Analyst"
        description: "Business reporting, dashboards, SQL analysis, BI tools"
        signals:
          titles: ["Data Analyst", "Business Analyst", "BI Analyst", "Reporting Analyst"]
          keywords: ["dashboards", "reporting", "Tableau", "Power BI", "Looker", "business intelligence"]
          anti_signals: ["dbt", "pipeline", "modeling layer"]
          
      analytics_engineer:
        label: "Analytics Engineer"
        description: "dbt, metrics layer, data modeling, semantic layer"
        signals:
          titles: ["Analytics Engineer", "Data Modeling Engineer"]
          keywords: ["dbt", "data modeling", "metrics layer", "semantic layer", "transformation"]
          disambiguate_from: ["data_analyst", "data_engineer"]
          
      data_engineer:
        label: "Data Engineer"
        description: "Pipelines, infrastructure, ETL/ELT, big data"
        signals:
          titles: ["Data Engineer", "ETL Developer", "Data Platform Engineer"]
          keywords: ["pipeline", "Airflow", "Spark", "ETL", "ELT", "data infrastructure", "Kafka"]
          
      ml_engineer:
        label: "ML Engineer"
        description: "Production ML systems, MLOps, includes LLM/GenAI implementation"
        signals:
          titles: ["ML Engineer", "Machine Learning Engineer", "MLOps Engineer", "AI Engineer"]
          keywords: ["model deployment", "MLOps", "feature store", "model serving", "LLM", "fine-tuning"]
          notes: "AI Engineer roles classify here if model-focused; out_of_scope if primarily API integration"
          
      data_scientist:
        label: "Data Scientist"
        description: "Statistical modeling, predictions, business insights"
        signals:
          titles: ["Data Scientist", "Senior Data Scientist", "Applied Scientist"]
          keywords: ["statistical modeling", "prediction", "regression", "classification", "causal inference"]
          disambiguate_from: ["ml_engineer", "research_scientist"]
          
      research_scientist:
        label: "Research Scientist (ML/AI)"
        description: "Novel ML research, publications, pushing state-of-the-art"
        signals:
          titles: ["Research Scientist", "ML Researcher", "AI Researcher"]
          keywords: ["publications", "novel", "state-of-the-art", "research", "PhD"]
          
      data_architect:
        label: "Data Architect"
        description: "Data strategy, governance, platform design"
        signals:
          titles: ["Data Architect", "Enterprise Data Architect", "Data Governance Lead"]
          keywords: ["data strategy", "governance", "data catalog", "metadata", "architecture"]

seniority:
  junior:
    label: "Junior"
    signals:
      titles: ["Junior", "Jr", "Associate", "Entry Level", "Graduate"]
      experience: ["0-2 years", "entry level", "new grad"]
      
  mid:
    label: "Mid-Level"
    signals:
      titles: ["Data Engineer", "Product Manager"] # No prefix = usually mid
      experience: ["2-5 years", "3+ years"]
      
  senior:
    label: "Senior"
    signals:
      titles: ["Senior", "Sr", "Lead"]
      experience: ["5+ years", "7+ years"]
      
  staff_plus:
    label: "Staff+"
    signals:
      titles: ["Staff", "Principal", "Distinguished", "Architect", "Director"]
      experience: ["10+ years", "extensive experience"]

working_arrangement:
  onsite:
    label: "Onsite"
    signals: ["on-site", "in-office", "office-based", "in-person"]
    
  hybrid:
    label: "Hybrid"
    signals: ["hybrid", "flexible", "2-3 days in office", "partial remote"]
    
  remote:
    label: "Remote"
    signals: ["remote", "work from home", "distributed", "anywhere"]
    qualifiers: ["remote (US only)", "remote (timezone restricted)"]

skills:
  parent_categories:
    product:
      label: "Product Skills"
      families:
        - discovery_research
        - execution_delivery
        - experimentation
        - analytics_pm
        - stakeholder_mgmt
        
    data_ml:
      label: "Data/ML Skills"
      families:
        - programming
        - analytics_stats
        - classical_ml
        - deep_learning
        - llm_genai
        - big_data
        - pipelines_orchestration
        - data_modeling
        - warehouses_lakes
        - mlops
        - cloud
        - streaming
        - visualization
        
    platform_infra:
      label: "Platform/Infra Skills"
      families:
        - deployment
        - infrastructure_code
        - ci_cd
        - monitoring

data_ml:
  programming:
    label: "Programming Languages"
    skills: ["Python", "R", "SQL", "Scala", "Java", "Julia"]
    notes: "SQL is both a language and a skill; always extract"
    
  analytics_stats:
    label: "Analytics & Statistics"
    skills: ["Statistics", "Probability", "Regression", "Causal inference", 
             "Time series", "Hypothesis testing", "Bayesian analysis"]
             
  classical_ml:
    label: "Classical Machine Learning"
    skills: ["Scikit-learn", "XGBoost", "LightGBM", "Random Forest",
             "Logistic regression", "SVM", "Feature engineering"]
             
  deep_learning:
    label: "Deep Learning"
    skills: ["PyTorch", "TensorFlow", "Keras", "Neural networks",
             "CNNs", "RNNs", "Computer vision", "NLP"]
             
  llm_genai:
    label: "LLM/GenAI"
    skills: ["LLMs", "Transformers", "GPT", "BERT", "Claude",
             "Prompt engineering", "RAG", "Vector databases",
             "LangChain", "Embeddings", "Fine-tuning"]
    notes: "Fast-evolving category; review quarterly"
    
  big_data:
    label: "Big Data Processing"
    skills: ["Spark", "PySpark", "Hadoop", "Hive", "Presto", "Flink"]
    
  pipelines_orchestration:
    label: "Pipelines & Orchestration"
    skills: ["Airflow", "Dagster", "Prefect", "Luigi",
             "Data pipelines", "ETL", "ELT"]
             
  data_modeling:
    label: "Data Modeling"
    skills: ["dbt", "Data modeling", "Dimensional modeling",
             "Star schema", "Data warehouse design"]
             
  warehouses_lakes:
    label: "Warehouses & Lakes"
    skills: ["Snowflake", "BigQuery", "Redshift", "Databricks",
             "Athena", "Delta Lake", "Data lake"]
             
  mlops:
    label: "MLOps"
    skills: ["MLflow", "Kubeflow", "Model serving", "Model monitoring",
             "Feature stores", "Model registry", "Weights & Biases"]
             
  cloud:
    label: "Cloud Platforms"
    skills: ["AWS", "GCP", "Azure", "S3", "EC2", "Lambda", "Cloud Functions"]
    
  streaming:
    label: "Streaming"
    skills: ["Kafka", "Kinesis", "Pub/Sub", "Real-time processing"]
    
  visualization:
    label: "Data Visualization"
    skills: ["Tableau", "Power BI", "Looker", "Metabase",
             "Plotly", "Matplotlib", "Seaborn"]

1. Check title patterns against known signals
2. Analyze job description for disambiguating keywords
3. Look at team/department placement
4. Consider required tools/skills
5. Apply "where would the practitioner self-identify?" test
6. If still ambiguous, document and classify to best fit
7. Flag for quarterly taxonomy review

Hierarchical classification
├── Fixed categories
├── Rule-based assignment
└── Exact match on signals

Semantic network
├── Entities with relationships
├── Embedding-based similarity
├── Natural language queries
└── Fuzzy matching with confidence

analytics_engineer:
  embedding_description: |
    An Analytics Engineer builds and maintains the data transformation 
    layer between raw data sources and analyst-ready datasets. They 
    typically work with tools like dbt to create reusable data models, 
    define business metrics in a semantic layer, and ensure data quality 
    through testing. They bridge the gap between Data Engineers who 
    build pipelines and Data Analysts who consume clean data.
    
    Related roles: Data Analyst, Data Engineer, BI Developer
    Key differentiator: Focuses on transformation and modeling, not 
    pipeline infrastructure or end-user dashboards.

relationships:
  is_a:
    description: "Hierarchical parent-child"
    example: "Analytics Engineer IS_A Data Role"
    
  related_to:
    description: "Conceptually similar, often confused"
    example: "Analytics Engineer RELATED_TO Data Analyst"
    
  requires_skill:
    description: "Role typically requires this skill"
    example: "Analytics Engineer REQUIRES_SKILL dbt"
    
  collaborates_with:
    description: "Roles that frequently work together"
    example: "Analytics Engineer COLLABORATES_WITH Data Scientist"
    
  progression_to:
    description: "Common career progression"
    example: "Data Analyst PROGRESSION_TO Analytics Engineer"

embedding_approach:
  model: "text-embedding-3-small" # or similar
  
  what_to_embed:
    - role descriptions (paragraph form)
    - skill descriptions
    - job posting titles + first 500 chars
    
  similarity_thresholds:
    high_confidence: 0.85
    medium_confidence: 0.70
    needs_review: 0.50
    
  use_cases:
    - "Find roles similar to Analytics Engineer"
    - "What skills are adjacent to dbt?"
    - "Candidates with X skills might fit Y roles"

Natural language queries the ontology should support:

"Show me roles that are like Data Scientist but more engineering-focused"
→ Returns: ML Engineer, Analytics Engineer

"What skills should a Data Analyst learn to become an Analytics Engineer?"
→ Returns: dbt, data modeling, SQL (advanced), Git

"Find companies where Analytics Engineers report to Engineering not Analytics"
→ Returns: [requires company org data]

"Which roles commonly transition to Product Management?"
→ Returns: Data Analyst, Product Analytics, Data Scientist

## Taxonomy Change Log

### [Date] - v1.X.X

**Added:**
- [New category/skill] - Rationale: [why needed]

**Changed:**
- [Category] - [What changed] - Rationale: [why]

**Removed:**
- [Category/skill] - Rationale: [why deprecated]

**Edge Cases Resolved:**
- [Role pattern] → Now classifies as [category]

**Open Questions:**
- [Unresolved issue for next review]

## Classification: [Job Title]

**Input:** [Raw title and key description excerpts]

**Decision:**
- Family: [product/data]
- Subfamily: [specific category]
- Seniority: [level]
- Confidence: [high/medium/low]

**Signals Found:**
- Title: [matching patterns]
- Keywords: [matching terms]
- Tools: [specific tools mentioned]

**Disambiguation Notes:**
[If edge case, explain reasoning]

**Flags:**
- [ ] Needs human review
- [ ] New pattern for taxonomy consideration

## Gap Analysis: [Date]

**Coverage Summary:**
- Total roles analyzed: [N]
- Successfully classified: [N] ([%])
- Out of scope: [N] ([%])
- Low confidence: [N] ([%])

**Emerging Patterns:**
| Pattern | Frequency | Suggested Action |
|---------|-----------|------------------|
| [New title pattern] | [N] | [Add category / Add signal / Monitor] |

**Problem Categories:**
| Category | Issue | Recommendation |
|----------|-------|----------------|
| [Category] | [High confusion rate with X] | [Improve signals / Merge / Split] |

**Skill Gaps:**
- [New skills appearing frequently but not in taxonomy]

**Recommendations:**
1. [Specific change]
2. [Specific change]

TAXONOMY_CONTEXT = """
Valid subfamilies for Data roles:
- product_analytics: Product metrics, experiments, user behavior
- data_analyst: Business reporting, dashboards, BI tools
- analytics_engineer: dbt, metrics layer, data modeling
- data_engineer: Pipelines, ETL/ELT, data infrastructure
- ml_engineer: Production ML, MLOps, model deployment
- data_scientist: Statistical modeling, predictions
- research_scientist: Novel ML research, publications
- data_architect: Data strategy, governance

Classification rules:
- "AI Engineer" → ml_engineer if model-focused, else out_of_scope
- Presence of "dbt" strongly indicates analytics_engineer
- "Business Analyst" → data_analyst unless product-focused
"""

-- User selects "Analytics Engineer" 
-- Only returns exact subfamily match, not "Data Analyst"
WHERE job_subfamily = 'analytics_engineer'

-- User selects "Data" family
-- Returns all data subfamilies
WHERE job_family = 'data'

# Current: exact match
results = db.query("subfamily = 'analytics_engineer'")

# Future: semantic similarity
query_embedding = embed("data transformation and metrics modeling role")
results = vector_search(query_embedding, threshold=0.8)
# Returns: analytics_engineer, data_engineer (lower score)