Firm-year analysis datasets

These products aggregate model scores or assemble outcomes at firm-year grain.

Blue nodes are upstream inputs, amber nodes are local support datasets, green nodes are reusable analysis-layer datasets, and purple nodes are terminal datasets read by analysis code. The dashed amber node is configured but not wired into the current build.

flowchart TB
    scores[(Worker-firm-period scores)]
    firms[(Firms)]
    rg[(Revelio-Compustat mapping)]
    comp[(Processed annual Compustat)]
    rgp[(Revelio-Compustat-PERMNO mapping)]
    patents_raw[(Raw KPSS patents)]
    returns_raw[(Raw CRSP monthly returns)]
    comp_extra[(Compustat analysis controls)]

    company_mq[(Company-year MQ)]
    mq_stats[(Firm-year MQ statistics)]
    rnd_mq[(R&D-worker MQ panel)]
    patent_panel[(Analysis patent panel)]
    volatility[(Returns volatility)]
    public_panel[(Public-firm analysis dataset)]

    scores --> company_mq
    firms --> company_mq
    rg --> company_mq
    comp --> company_mq

    scores --> mq_stats
    scores -.-> rnd_mq
    patents_raw --> patent_panel
    rgp --> patent_panel
    returns_raw --> volatility

    company_mq --> public_panel
    mq_stats --> public_panel
    patent_panel --> public_panel
    volatility --> public_panel
    rgp --> public_panel
    comp_extra --> public_panel

    classDef input fill:#e8eef7,stroke:#52739e,stroke-width:2px
    classDef support fill:#fff0cc,stroke:#a66b00,stroke-width:2px
    classDef reusable fill:#e2f2e7,stroke:#3b7a4b,stroke-width:2px
    classDef analysis fill:#eee5f5,stroke:#76528c,stroke-width:2px
    classDef dormant fill:#fff0cc,stroke:#a66b00,stroke-width:2px,stroke-dasharray:5 5
    class scores,firms,rg,comp,rgp,patents_raw,returns_raw,comp_extra input
    class mq_stats,patent_panel,volatility support
    class company_mq reusable
    class public_panel analysis
    class rnd_mq dormant

Datasets and roles

Dataset	Role	Grain
Company-year MQ	Reusable across analysis-data pipelines	Company-year
Firm-year MQ statistics	Local public-firm support	Company-year
Analysis patent panel	Local public-firm support	Security-year
Returns volatility	Local public-firm support	Security-year
Public-firm analysis dataset	Direct analysis input	Public firm-year
rnd_mq_panel.parquet	Configured but disconnected local support	Company-year

Overlapping company-year MQ aggregations

company_year_mq.parquet and firm_mq_stats.parquet independently read the worker-firm-period scores and aggregate them by company and year. Both store the observation count and mean MQ under different column names; firm_mq_stats.parquet additionally stores variance and median. The public-firm builder then joins the two products. This is a real duplication in the current pipeline, not merely a documentation distinction. A third builder, build_rnd_mq_panel.R, separately aggregates MQ for engineering and science workers, but no build entry point invokes it and it does not feed another dataset in the current pipeline.

The public-firm performance and innovation page documents that branch in more detail, including its raw patent and returns inputs.