I still have most of the Halloween candy I bought this year. We live within walking distance of three public schools. But where are the kids? Very few were in our neighborhood trick-or treating. In fact, not many families with young children live here anymore.

This is partially on us.

According to a recent CNN report: “Fewer homeowners selling is part of what is keeping the inventory of homes historically low and pushing prices ever higher in markets across the U.S.” Empty nesters own more home than they need, while millennials with kids can’t find or afford homes that can reasonably accommodate a family.

Some baby boomers who are not selling their family homes say they need the space to host their adult children and grandchildren — that’s us. Others would like to downsize but can’t find what they need or don’t think a smaller place is worth the money they’d have to spend.

Capital gains taxes are keeping millions of homes off the market. A third of homeowners have made enough on their homes that they’d owe federal taxes if they sold. They’re staying put, so their heirs can inherit the property with a stepped-up tax basis and avoid capital gains.

A Washington Post article states that “empty nesters’ economics” are creating a “market-wide logjam.” Jim Parrott of the Urban Institute writes: “A scarcity of available housing in high-growth job markets means fewer workers can relocate to areas with greater opportunity, leaving jobs unfilled and underemployed workers stuck in the wrong places.”

Building more small homes would help. Realtor.com laments “The Death of the Starter Home.” They point out that builders find it more lucrative to put up larger homes. So millennials, urban renters, downsizing boomers, and investors are vying for the dwindling supply of smaller houses.

This lack of housing serves as a deterrent to family formation and growth. We want young adults to marry and have children. They’ll need suitable homes to live in.