Regarding the Confederate Flag?

Why do people condemn the Confederate Flag when it is used, (mis-used), by racist? These same racist fly the US flag don't consider that flag to be racist. The US flag flew over slavery much longer than the Confederate flag did. The Confederate Soldiers were American soldiers, (declared so by Congress).