{ "id": "epc5pcz3n8", "slug": "john-schulman", "title": "John Schulman", "updated": "2026-10-15", "topic_path": [ { "id": "technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/topics/technology" }, { "id": "technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing" }, { "id": "technology.computing.modern_ai", "label": "Modern AI: foundation models, generative AI, and the AI industry", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.modern_ai" }, { "id": "technology.computing.modern_ai.industry", "label": "AI companies, people, and products", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.modern_ai.industry" }, { "id": "technology.computing.modern_ai.industry.ai_founders_executives", "label": "AI founders and executives", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.modern_ai.industry.ai_founders_executives" }, { "id": "technology.computing.modern_ai.industry.ai_founders_executives.applied-ai-startup-founders", "label": "Applied AI startup founders", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.modern_ai.industry.ai_founders_executives.applied-ai-startup-founders" } ], "geo": [ { "id": "geo.us.t2001.technology.computing.modern_ai.industry.ai_founders_executives", "label": "United States · 2001 to 2020: AI founders and executives", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2001.technology.computing.modern_ai.industry.ai_founders_executives", "path": [ { "id": "geo.us", "label": "United States", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us" }, { "id": "geo.us.t2001", "label": "United States · 2001 to 2020", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2001" }, { "id": "geo.us.t2001.technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2001.technology" }, { "id": "geo.us.t2001.technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2001.technology.computing" }, { "id": "geo.us.t2001.technology.computing.modern_ai", "label": "Modern AI: foundation models, generative AI, and the AI industry", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2001.technology.computing.modern_ai" }, { "id": "geo.us.t2001.technology.computing.modern_ai.industry", "label": "AI companies, people, and products", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2001.technology.computing.modern_ai.industry" }, { "id": "geo.us.t2001.technology.computing.modern_ai.industry.ai_founders_executives", "label": "AI founders and executives", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.us.t2001.technology.computing.modern_ai.industry.ai_founders_executives" } ] } ], "excerpt": "John Schulman is an American reinforcement-learning researcher, a co-founder of OpenAI, and creator of the PPO algorithm who helped build ChatGPT before leaving for Anthropic in 2024.", "snippet": "John Schulman is an American reinforcement-learning researcher, a co-founder of OpenAI, and creator of the PPO algorithm who helped build ChatGPT before leaving for Anthropic in 2024.", "node": "technology.computing.modern_ai.industry.ai_founders_executives.applied-ai-startup-founders", "markdown": "# John Schulman\n\n**John Schulman** is an American reinforcement-learning researcher, a co-founder of OpenAI, and the creator of the [Proximal Policy Optimization](https://www.edgepedia.org/proximal-policy-optimization) (PPO) algorithm, one of the most widely used methods in reinforcement learning. He helped build ChatGPT and the reinforcement learning from human feedback (RLHF) pipeline behind it, left OpenAI for Anthropic in August 2024, and later became chief scientist of [Mira Murati](https://www.edgepedia.org/mira-murati)'s Thinking Machines Lab.[1](https://theinfoweb.org/people/john-schulman/)\n\n| Key fact | Detail |\n|---|---|\n| Education | Physics at Caltech; PhD in robotics and reinforcement learning at UC Berkeley under Pieter Abbeel[2](https://frontierminds.ai/people/john-schulman/) |\n| TRPO | Trust Region Policy Optimization, ICML 2015, with Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz; optimizes control policies with guaranteed monotonic improvement[3](https://proceedings.mlr.press/v37/schulman15) |\n| PPO | Proximal Policy Optimization, July 2017, with Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov; TRPO's benefits with first-order optimization only[4](https://arxiv.org/abs/1707.06347) |\n| RLHF at OpenAI | Core technique behind InstructGPT and ChatGPT; the InstructGPT reward model learned from comparisons on about 33,000 prompts[5](https://www.rlresearch.ai/methodologies/rlhf/) • [6](https://theisnn.com/the-inside-story-of-how-chatgpt-was-built-from-the-people-who-made-it/) |\n| Career moves | Left OpenAI for Anthropic in August 2024; left Anthropic in February 2025; cofounded Thinking Machines Lab as chief scientist[1](https://theinfoweb.org/people/john-schulman/) • [2](https://frontierminds.ai/people/john-schulman/) |\n| Other work | Generalized Advantage Estimation (2015); proof that soft (entropy-regularized) Q-learning is exactly equivalent to a policy gradient method (2017)[7](https://arxiv.org/abs/1506.02438) • [8](https://arxiv.org/abs/1704.06440) |\n\n## Education and the founding of OpenAI\n\nSchulman studied physics at Caltech and then did a PhD in robotics and reinforcement learning at the [University of California](https://www.edgepedia.org/university-of-california), Berkeley, under [Pieter Abbeel](https://www.edgepedia.org/pieter-abbeel), a Berkeley professor known for work in robot learning.[2](https://frontierminds.ai/people/john-schulman/) His dissertation, *Optimizing Expectations: From Deep Reinforcement Learning to Stochastic Computation Graphs*, covers the line of work that produced TRPO and variance-reduction methods for policy gradients.[9](https://escholarship.org/uc/item/9z908523)\n\nHe was part of the group that founded OpenAI. According to a Time profile of [Sam Altman](https://www.edgepedia.org/sam-altman), Altman drove to Berkeley to go for a walk with graduate student John Schulman, then had dinner with [Greg Brockman](https://www.edgepedia.org/greg-brockman) and met [Wojciech Zaremba](https://www.edgepedia.org/wojciech-zaremba); OpenAI launched in December 2015.[10](https://time.com/6342827/ceo-of-the-year-2023-sam-altman/)\n\n## Trust Region Policy Optimization\n\nDuring and shortly after his PhD, Schulman developed [Trust Region Policy Optimization](https://www.edgepedia.org/trust-region-policy-optimization) (TRPO) with [Sergey Levine](https://www.edgepedia.org/sergey-levine), Pieter Abbeel, Michael Jordan, and Philipp Moritz. Published at ICML 2015, the method optimizes control policies with guaranteed monotonic improvement: it restricts how much a single update is allowed to change the policy, making large nonlinear policies such as neural networks trainable reliably.[3](https://proceedings.mlr.press/v37/schulman15) • [11](https://arxiv.org/abs/1502.05477) TRPO is a second-order algorithm that solves an approximation of a constrained optimization problem using conjugate gradient.[12](https://openreview.net/pdf?id=ByOnmlWC-) It performed well on simulated robotic locomotion and on Atari games played from screen images.[9](https://escholarship.org/uc/item/9z908523)\n\nA companion 2015 paper with Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel, Generalized Advantage Estimation (GAE), attacks the variance of policy-gradient estimates using a state-value function, a improvement complementary to TRPO's trust-region constraint.[7](https://arxiv.org/abs/1506.02438) • [9](https://escholarship.org/uc/item/9z908523)\n\n## Proximal Policy Optimization\n\nIn July 2017 Schulman and OpenAI colleagues Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov published Proximal Policy Optimization. PPO alternates between sampling data from the environment and optimizing a surrogate objective with stochastic gradient ascent, and its novel objective permits multiple epochs of minibatch updates on the same data, where standard policy gradient methods perform one gradient update per sample.[4](https://arxiv.org/abs/1707.06347) The paper's stated goal was to keep the data efficiency and reliable performance of TRPO while using only first-order optimization.[13](https://rainboltz.github.io/rlfxer/ppo.pdf)\n\nPPO replaces TRPO's hard constraint with a clipping mechanism, so a first-order optimizer such as gradient descent suffices.[14](https://proceedings.mlr.press/v115/wang20b/wang20b.pdf) This makes it simpler to implement, more general, and compatible with architectures that share parameters between a policy and value function or include noise such as dropout, which TRPO's conjugate-gradient machinery does not handle; PPO also works naturally with stochastic computation graphs.[4](https://arxiv.org/abs/1707.06347) • [13](https://rainboltz.github.io/rlfxer/ppo.pdf) • [15](https://www.dooperator.ai/research/papers/edf73203-2ace-4ebc-a1b8-2aff2909df97) On benchmarks including simulated robotic locomotion and Atari games, PPO outperformed other online policy gradient methods and struck a favorable balance between sample complexity, simplicity, and wall-clock time.[4](https://arxiv.org/abs/1707.06347) OpenAI's Baselines release credits the algorithm to Schulman, Klimov, Wolski, Dhariwal, and Radford, and notes that ACER, the main alternative, is far more complicated, requiring off-policy corrections and a replay buffer while doing only marginally better on Atari.[16](https://openai.com/index/openai-baselines-ppo/)\n\nPPO nonetheless became the default RL algorithm in many labs and was the optimizer in InstructGPT-style RLHF.[17](https://www.roadtointelligence.com/concepts/rlhf/)\n\n## RLHF and ChatGPT\n\nAt OpenAI, Schulman led the alignment work that turned language models into instruction-following assistants. The technique, reinforcement learning from human feedback, trains a reward model from human comparisons and then optimizes the policy against it with PPO. InstructGPT, the template for ChatGPT, learned its reward model from comparisons on about 33,000 prompts, with labelers ranking several answers per prompt.[5](https://www.rlresearch.ai/methodologies/rlhf/) ChatGPT was trained in a very similar way to [InstructGPT](https://www.edgepedia.org/instructgpt) using RLHF; Schulman explained in a 2023 interview that the raw benchmark capabilities of the models did not differ substantially, but ChatGPT was more accessible and usable.[6](https://theisnn.com/the-inside-story-of-how-chatgpt-was-built-from-the-people-who-made-it/)\n\n## Other research\n\nWith [Xi Chen](https://www.edgepedia.org/xi-chen) and Pieter Abbeel, Schulman showed in 2017 that there is a precise equivalence between [Q-learning](https://www.edgepedia.org/q-learning) and policy gradient methods in entropy-regularized reinforcement learning: \"soft\" Q-learning is exactly equivalent to a policy gradient method, which helps explain why Q-learning methods sometimes behave like policy gradient updates.[8](https://arxiv.org/abs/1704.06440)\n\n## Later career\n\nHe left [Anthropic](https://www.edgepedia.org/anthropic) in February 2025 and became chief scientist of [Thinking Machines Lab](https://www.edgepedia.org/thinking-machines-lab), the startup founded by former OpenAI chief technology officer Mira Murati.[1](https://theinfoweb.org/people/john-schulman/) • [2](https://frontierminds.ai/people/john-schulman/)\n\n## References\n\n1. [John Schulman, The Info Web](https://theinfoweb.org/people/john-schulman/)\n2. [John Schulman, Thinking Machines Lab, Frontier Minds](https://frontierminds.ai/people/john-schulman/)\n3. [Trust Region Policy Optimization, Schulman et al., ICML 2015, PMLR v37](https://proceedings.mlr.press/v37/schulman15)\n4. [Proximal Policy Optimization Algorithms, Schulman et al., arXiv:1707.06347](https://arxiv.org/abs/1707.06347)\n5. [RLHF, RL Research](https://www.rlresearch.ai/methodologies/rlhf/)\n6. [The inside story of how ChatGPT was built, THE ISNN](https://theisnn.com/the-inside-story-of-how-chatgpt-was-built-from-the-people-who-made-it/)\n7. [High-Dimensional Continuous Control Using Generalized Advantage Estimation, arXiv:1506.02438](https://arxiv.org/abs/1506.02438)\n8. [Equivalence Between Policy Gradients and Soft Q-Learning, arXiv:1704.06440](https://arxiv.org/abs/1704.06440)\n9. [Optimizing Expectations: From Deep Reinforcement Learning to Stochastic Computation Graphs, eScholarship](https://escholarship.org/uc/item/9z908523)\n10. [Sam Altman, CEO of the Year 2023, Time](https://time.com/6342827/ceo-of-the-year-2023-sam-altman/)\n11. [Trust Region Policy Optimization, arXiv:1502.05477](https://arxiv.org/abs/1502.05477)\n12. [ICLR 2018 conference paper discussing TRPO and PPO, OpenReview](https://openreview.net/pdf?id=ByOnmlWC-)\n13. [Proximal Policy Optimization Algorithms (paper copy)](https://rainboltz.github.io/rlfxer/ppo.pdf)\n14. [Truly Proximal Policy Optimization, Wang et al., PMLR v115](https://proceedings.mlr.press/v115/wang20b/wang20b.pdf)\n15. [Proximal Policy Optimization Algorithms, DoOperator Research](https://www.dooperator.ai/research/papers/edf73203-2ace-4ebc-a1b8-2aff2909df97)\n16. [Proximal Policy Optimization, OpenAI Baselines](https://openai.com/index/openai-baselines-ppo/)\n17. [Reinforcement Learning from Human Feedback, Road to Intelligence](https://www.roadtointelligence.com/concepts/rlhf/)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Modern AI: foundation models, generative AI, and the AI industry › AI companies, people, and products › AI founders and executives › Applied AI startup founders*\n\n*Initially written Sep 17, 2026 · Reviewed: Sep 30, 2026; Oct 15, 2026 · Edited: Sep 30, 2026; Oct 11, 2026; Oct 15, 2026 · Last review: Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [], "url": "https://www.edgepedia.org/john-schulman", "markdown_url": "https://www.edgepedia.org/john-schulman.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"John Schulman\", Edgepedia (EdgeChat), https://www.edgepedia.org/john-schulman. Edgepedia Community License 1.0.", "credit_md": "\"[John Schulman](https://www.edgepedia.org/john-schulman)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/john-schulman](https://www.edgepedia.org/john-schulman). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"John Schulman\", Edgepedia (EdgeChat), https://www.edgepedia.org/john-schulman. Edgepedia Community License 1.0.", "speakable": "John Schulman is an American reinforcement-learning researcher, a co-founder of OpenAI, and creator of the PPO algorithm who helped build ChatGPT before leaving for Anthropic in 2024." }