SpeechMap.AI compares language models’ refusal rates on controversial speech topics.
Automated Social Science: Language Models as Scientist and Subjects NBER
This NBER paper studies language models as both researchers and subjects in automated social science.
About Automated Social Science: Language Models as Scientist and Subjects NBER
This NBER Working Paper 32381 (April 2024) by Benjamin S. Manning, Kehang Zhu and John J. Horton describes an approach for automatically generating and testing social science hypotheses in silico. Large language models serve as both researcher and subjects, while structural causal models provide the language for hypotheses, the blueprint for LLM-based agents, the experimental design and the data analysis plan.
The paper demonstrates the method in negotiation, bail hearing, job interview and auction scenarios, where causal relationships are proposed and tested. It reports that simulation insights were not available through direct elicitation alone, and that the fitted structural causal model can be used for prediction or planning follow-on experiments.
Key features
- Automatically generates social science hypotheses in silico
- Tests hypotheses using LLM-based agents
- Uses structural causal models to state hypotheses
- Provides experimental design and data analysis plan
- Produces fitted model for prediction and follow-on experiments
- Demonstrates method across negotiation, bail, interview, auction scenarios
Best for: Researchers interested in automated hypothesis generation and testing with language models.
Good for
- Tested in four scenarios: negotiation, bail hearing, job interview and auction
- In silico auction results closely matched predictions of auction theory
- Conditioning on fitted model improved LLM prediction of outcomes
Watch out for
- LLM could predict signs of effects but not reliably predict magnitudes
- Directly elicited clearing-price predictions were inaccurate in auction test
Automated Social Science: Language Models as Scientist and Subjects NBER at a glance
- Website
- nber.org/papers/w32381
- Category
- AI & Machine Learning › AI Data & Analytics
- Type
- Website
- Pricing
- Not confirmed. Check the site
- Platforms
- Web
- Open source
- Not stated
- Language
- English
- Last checked
- Oct 5, 2026 · Online when checked
- Listed
- Oct 5, 2026
Automated Social Science: Language Models as Scientist and Subjects NBER alternatives
Tools and sites you could use instead, ranked by how closely they match.
OpenLM Arena compares open language models through leaderboards and benchmark results.
VoxelBench benchmarks language models on creating voxel builds from prompts.
KNIME provides data analysis tools and a visual workflow builder for data science solutions.
The Hugging Face Open LLM Leaderboard compares language models across evaluation tests including IFEval, MATH, and MMLU-Pro.
FastDatasets helps create training datasets for large language models.
Open source
24 alternativesPopular in AI Data & Analytics
Well-known listings from the same topic.
Dotgetdot.aiForvioforvio.comFluidAI Medicalfluidai.mdApache Zeppelinzeppelin.apache.orgSola Securitysola.securityQuadraticquadratichq.com
Automated Social Science: Language Models as Scientist and Subjects NBER FAQ
What scenarios are used to demonstrate the approach?
A negotiation, a bail hearing, a job interview, and an auction.
What is the role of structural causal models in the approach?
They provide a language to state hypotheses, a blueprint for constructing LLM-based agents, an experimental design, and a plan for data analysis.
Who wrote the paper and when was it issued?
Benjamin S. Manning, Kehang Zhu and John J. Horton; Working Paper 32381, April 2024.
Is something wrong or out of date?
Tell us and an editor will check it.
Own this website?
Claim this listing
Claim the listing and suggest updates yourself. An editor checks every claim first.