The scriptures do not give elders authority. Elders are commanded to watch over people, serve them and protect them. But they are not given authority to control them.
Because we are engaged in a spiritual battle, new Jesus-followers are encouraged to submit to elders. This free action gives authority to the elder (an elder does not have an innate authority). They only have authority that is created when someone submits to them. If the people submitting lose their respect for the elder and withdraw their submission, the elder loses their authority.
Elders often demand that people in their church submit to them. This is a false understanding of authority. If people must obey them, they are exercising domination and control, which is a distortion of true authority. True authority is given by freely by submission, it cannot be demanded.
The same applies to husbands and wives. Wives are urged to submit to their husbands (just as husbands have to submit to the will of their wives, because they love them). Husbands do not have innate authority that allows them to demand that their wives obey them. Demanded obedience is actually domination.
Submission that cannot be freely withdrawn at any time, is domination and control, not true authority.
Christians should stop claiming that they have authority, because they are male or mature, or have been appointed to a particular role. They should earn authority, by loving, serving and demonstrating wisdom.