Houston-based Marathon Oil Co. agreed to invest $177 million in extensive compliance measures—a condition of a settlement reached Thursday with the Environmental Protection Agency and the Justice Department.