I have a question about a small fire (in a fireplace)?

When I have a big fire going, the room gets hotter. But when it dies down just to embers, the rooms seems to be colder than it was before I lit the fire. I though about it and wondered if it was anything to do with the hot embers sucking the hot air out of the room, up the chimney?