Lets say the Japanese caught both US carriers in port and they did destroy the fleet oil storage tanks and repair facilities at Pearl Harbor - would it have made much of a difference? Nope. Maybe a six to twelve month extension on the war but the end result would have been the same.
Here's my data source:
http://www.combinedfleet.com/economic.htm