I use reduced order models (ROMs) to substantially decrease the computational cost of Newton's method for large-scale time-dependent optimal control problems in settings where solving the implicit constraints and their adjoints can be prohibitively expensive. Reservoir management in particular can take weeks to solve the state and adjoint equations necessary to generate one gradient evaluation. Model order reduction has potential to reduce the cost; however, ROMs are valid only nearby their training and must be retrained as the optimization progresses. I will demonstrate that in the case of reservoir management, frequent retraining defeats of purpose the reduced order modeling the state equation altogether. Rather than generating a reduced order model to replace the original implicit constraint, I use the structure of Hessian and subspace-based ROMs to compute approximate reduced order Hessian information by recycling data from the full order state and adjoint solves in the gradient computation. The resulting method often enjoys convergence rates similar to those of Newton's Method, but at the computational cost of the gradient method. I demonstrate my approach on nonlinear parabolic optimal control problems on two cases: (1) a semilinear parabolic case with nonlinear reaction terms and (2) the well rate optimization problem in reservoir engineering. For semilinear parabolic case, I consider a cubic reaction term and an exponential reaction term modeling solid fuel injection. My results show a dramatic speedup in wall clock time for the entire optimization procedure. This overall acceleration includes the training and precomputation of the ROMs that is conventionally discounted as "offline" computational costs.