Climate change is expected to reduce agricultural productivity in most developing countries. Classic international trade and geography models predict that the optimal adaptation response is a reallocation of capital and labor from agriculture towards other sectors and regions gaining comparative advantage. In this paper, we provide direct evidence on the effects of recent changes in climate in Brazil to understand to what extent factor market frictions constrain this reallocation process. First, we document that when a region experiences a temporary drought, the agricultural sector obtains insurance through capital flows coming from financially integrated regions. However, if droughts become persistent, we observe a sharp reduction in credit to all sectors in both drying regions and financially integrated regions. Second, persistent increases in dryness generate a large reduction in agricultural employment. Some workers reallocate towards local manufacturing and others out-migrate, as predicted by classical models. In contrast, climate migrants are mostly allocated to small firms outside of manufacturing in destination regions.