This can happen if there are some infrastructure glitches with the caller setup. Not just that, on your side, please be sure that the in-ear speaker volume level is adequate/adjusted:
While you're on a call, by using the volume buttons you can adjust. They're located on the side of the iPhone/iPad, and the top button makes calls louder in volume and the bottom button makes calls quieter.
You may need to raise your call volume if you're in a noisy environment or the person you're speaking to is speaking softly.