"If you have a large enough population" is key to this. Large enough (to geneticists) may mean in the thousands, or at least hundreds.
Backing up a bit.You all know that genes come in pairs on chromosomes, right? The location of a specific gene (call it G) on its chromosome pair is called its locus (plural=loci). So every individual has two Gs. There may be several or many different versions (called alleles) of G1 in a species; call them G1, G2, etc. but each individual has exactly two Gs. One individual may be of genotype(=genetic makeup) G1G1, another G1G5, another G4G3. Lots of possibilities. If, through selection and breeding, you've managed to get rid of all alleles except the one you want, all individuals in your population will be (say) G2G2, and the allele/trait is said to be "fixed" in the population. Thing is, that when you select, inbreed, backcross, sib mate, all those things, in an effort to fix alleles at one locus, you reduce diversity not only at the locus you're interested in, but at others as well. This may or may not seem intuitively obvious to you. There's a lot of math involved in proving this, but the math has been well done, well understood, and well accepted for decades.
If through inbreeding you accidentally lose valuable alleles at other loci, you risk damaging the adaptive capability of your population.
Some alleles are present at very low frequencies, e.g. one in 10,000, one in a million. This is due partly to chance, and partly to the selective advantage conferred by the mutuation that gave rise to that allele. Some alleles may have no selective advantage at present, but may provide genetic resistance immunity to some as-yet-unknown disease or environmental challenge in the future. Even alleles that are present at relatively high frequencies (say one in 50) are at very high risk of loss over even one or two generations in a breeding population of only 25. Thus the value of keeping diversity for diversity's sake.