In order to dissolve in water, a chemical must be able to create ions, which are molecules with partially positive and partially negative charges.
Table salt (NaCl), for example, is composed of sodium and chloride ions. Both of these ions can dissolve in water because the positive dipole of water molecules attracts chloride and the negative dipole of water molecules attracts positive sodium ions.
This causes dissolving by breaking the crystal structure of NaCl.
Chalk is mostly made up of limestone (calcium carbonate), with some silt and clay thrown in for good measure. Silt and clay do not dissolve in water, as we know.