Dante is being abusive here, and the narrator does recognize it, yet cannot resist anymore. By now he’s distressed by hell and the realization that he is also guilty of wrongdoing by association and by ignorance, feeling responsible for all the evils of the world.
Surely Dante’s being unfair, and his reasoning is unsound. He berates the pilgrim for wanting some solace, equating such a wish for wanting to take drugs. And he blames the pilgrim for wanting to make him a kind of drug peddler when in fact he has much more noble things in mind. Even if intellectual, poetic or religious pursuits are just as much illusions as chasing dragons in drug-fuelled haze, he states it is still more noble to dedicate your life to scriptures and arts.
It doesn’t sound very convincing, and the pilgrim is starting to question his stance, if not his authority. As much as the pilgrim loves Dante, he’s also suffered enough by now to be afraid of angering him further. In the end the narrator does manage to say that he’s already broken and cannot stand his abuse, and Dante does apologize. Yet for the first time cracks are showing in their relationship, some permanent difference in worldview. Dante’s faith in what he pursued is absolute, whereas the pilgrim comes from a modern world which is much more about questioning our values and morals, whatever they are.