The Internet of Things (IoT) is changing the way we interact with our environment in domains as diverse as health, transportation, office buildings and our homes. In smart building environments, information captured about the building and its inhabitants will aid in development of services that improve productivity, comfort, social interactions, safety, energy savings and more. However, by collecting and sharing information about building’s inhabitants and their activities, these services also open the door to privacy risks. In this paper, we introduce a framework where IoT Assistants capture and manage the privacy preferences of their users and communicate them to privacy-aware smart buildings, which enforce them when collecting user data or sharing it with building services. We outline elements necessary to support such interactions and also discuss important privacy policy attributes that need to be captured. This includes looking at attributes necessary to describe – (1) the data collection and sharing practices associated with deployed sensors and services in smart buildings as well as (2) the privacy preferences to help users manage their privacy in such environments.