In my experience as a swim coach, I have never had an athlete sit down with me to discuss goals and say "I just want to be average". However, I've seen too many athletes paving the path to mediocrity daily during practices. In my last stint of goal meetings, I've asked athletes to think beyond their goals, but to also define, and write down, what they are actually willing to do to achieve them. I've called it an exercise of honesty with themselves.
When people marvel at accomplishments, the path it took to get there is too often overlooked. After Beijing, what made me admire Phelps' accomplishments was not so much the 8 golds, but fact that, according to all accounts, he had practiced all 1,461 days in the 4 years leading to it! The background stories on projects Steve Jobs headed show the same type of intense, relentless dedication to minute details. Basically, there are no great accomplishments without a background of true dedication toward a goal.
Back to swimming, if your answer to this post's title is no, then you need break from the pack, and practice daily according to what you want to achieve. If being mediocre is not your goal (and again, I never heard anyone with that goal), then you need put in quality work that matches your goals! While there are no guarantees to fast swimming and achieving goals, there are ways to increase the likelihood of those happening. This includes quality, focused drills, attention to details (turns, breakouts, etc), embracing and looking forward to pain during hard sets and dryland, and a positive attitude about doing it all again the next day!
How many are willing to practice daily according to the standards they set to themselves? I hope you are one of them!