With the rapid adoption of smart speakers in people’s homes, there is a corresponding increase in users’ privacy and security concerns. In contrast to previous studies of users’ concerns about smart speakers’ divulging private information to their manufacturers, our study focused on investigating users’ concerns with regard to housemates and external entities. We conducted semi-structured interviews with 26 participants living in 21 households. Our results suggest that users often have an inadequate understanding of what data their smart speakers makes available to all users and what is kept private. Although participants expressed different privacy concerns about their housemates and external entities, they adopted similar, yet suboptimal, risk management strategies. We provide recommendations for future speaker design to support more optimal coping with the perceived risks.