Tunneling is essential in the initialization, measurement, and control of quantum dot qubits. In silicon, such tunneling connects not only the qubit states but also valley minima in the conduction band on opposite sides of the Brillouin zone, with large consequences for the quantum dot behavior. Here we present a full characterization of the intravalley and intervalley tunnel couplings, including their complex phases -- the valley phases. These phases are shown to control measurable parameters, including the ratios of the gaps at anticrossings between quantum states of a double quantum dot. The valley phases themselves evolve as a function of the quantum dot gate voltages and depend on the underlying atomic structure of the quantum well. Knowledge of the valley phases completes the picture and fills a key gap in our understanding of sample-wide variations of valley couplings and the physical parameters that depend on them, including spin-orbit coupling, valley-orbit mixing, and Landé $g$-factors.