Geometric and electromagnetic response and anomalies in topological phases of matter