A NaN is returned (under IEEE 754) in order to continue a computation when a truly undefined (intermediate) result has been obtained. An infinity is returned in order to continue a computation after an overflow has occurred.
Thus the behaviour
Math.sqrt(Double.NEGATIVE_INFINITY); // NaN
is specified because it is known (easily and quickly) that an undefined value has been generated; based solely on the sign of the argument.
However evaluation of the expression
Math.pow(Double.NEGATIVE_INFINITY, 0.5); // Infinity
encounters both an overflow AND an invalid operation. However the invalid operation recognition is critically dependent on how accurate the determination of the second argument is. If the second argument is the result of a prior rounding operation, then it may not be exactly 0.5. Thus the less serious determination, recognition of an overflow, is returned in order to avoid critical dependence of the result on the accuracy of the second argument.
Additional details on some of the reasoning behind the IEEE 754 standard, including the reasoning behind returning flag values instead of generating exceptions, is available in