GATE DA — Section 3

Calculus & Optimization

Comprehensive study notes on single-variable functions, limits, continuity, differentiability, Taylor series, and optimization for GATE Data Science & AI.

Official GATE DA Syllabus: Functions of a single variable, limit, continuity and differentiability, Taylor series, maxima and minima, optimization involving a single variable.

1. Functions of a Single Variable: Domains & Properties

A real-valued function of a single variable is a mapping f: D → ℝ where D ⊆ ℝ is the domain. In machine learning, loss functions, activation curves (e.g., Sigmoid, ReLU, GELU), and likelihood functions are single-variable slices along optimization coordinates.

  • Monotonicity: If f'(x) ≥ 0 on an interval I, f(x) is monotonically non-decreasing. If f'(x) > 0, it is strictly increasing and invertible.
  • Convexity: A function f(x) is strictly convex on interval I if for all x_1, x_2 ∈ I and λ ∈ (0, 1): f(λ x_1 + (1-λ)x_2) ≤ λ f(x_1) + (1-λ)f(x_2). Equivalently, if f''(x) ≥ 0 everywhere, every local minimum is guaranteed to be a global minimum.

2. Limits, Continuity & Differentiability

Understanding limits is crucial for evaluating gradient behavior and asymptotic stability:

  • Limit Existence: lim_{x→a} f(x) = L if and only if both left-hand and right-hand limits exist and coincide: lim_{x→a^-} f(x) = lim_{x→a^+} f(x) = L.
  • Continuity: A function f is continuous at x = a if:
    1. f(a) is defined.
    2. lim_{x→a} f(x) exists.
    3. lim_{x→a} f(x) = f(a).
  • Differentiability: f is differentiable at a if the derivative limit exists:
    f'(a) = lim_{h → 0} [f(a + h) - f(a)] / h
  • The Fundamental Theorem: Differentiability implies continuity. The converse is false: f(x) = |x| is continuous at x = 0, but has left derivative -1 and right derivative +1, so it is not differentiable at x = 0.

3. L'Hôpital's Rule & Indeterminate Forms

When evaluating limits producing indeterminate forms 0/0 or ∞/∞, where f and g are differentiable in an open interval around a (except possibly at a) and g'(x) ≠ 0:

lim_{x → a} [f(x) / g(x)] = lim_{x → a} [f'(x) / g'(x)]

For exponential indeterminate forms like 1^∞, 0^0, or ∞^0, convert using natural logarithms: y = f(x)^{g(x)} ⇒ ln y = g(x) ln f(x).

4. Taylor Series & Polynomial Approximation

Taylor series expansion allows us to approximate complex, non-linear objective functions with polynomials around an operating point x = a:

f(x) = f(a) + f'(a)(x - a) + \frac{f''(a)}{2!}(x - a)^2 + \dots + \frac{f^{(n)}(a)}{n!}(x - a)^n + R_n(x)

When expanded around a = 0, it is called a Maclaurin series. Key expansions to memorize for GATE:

  • e^x = 1 + x + \frac{x^2}{2!} + \frac{x^3}{3!} + \dots
  • ln(1 + x) = x - \frac{x^2}{2} + \frac{x^3}{3} - \dots \quad \text{for } -1 < x ≤ 1
  • sin(x) = x - \frac{x^3}{3!} + \frac{x^5}{5!} - \dots
  • cos(x) = 1 - \frac{x^2}{2!} + \frac{x^4}{4!} - \dots
  • \frac{1}{1 - x} = 1 + x + x^2 + x^3 + \dots \quad \text{for } |x| < 1

5. Maxima, Minima & Single-Variable Optimization

Single-variable optimization is the foundation for gradient descent step-size tuning and regularizer parameter search:

  • Fermat's Theorem (First Derivative Test): If f(x) has a local extremum at an interior point c and is differentiable, then f'(c) = 0. Points where f'(c) = 0 or where f'(c) is undefined are called critical points.
  • Second Derivative Test: Let c be a critical point where f'(c) = 0:
    • If f''(c) > 0, f(x) is strictly concave up → Local Minimum.
    • If f''(c) < 0, f(x) is strictly concave down → Local Maximum.
    • If f''(c) = 0, the test is inconclusive (requires higher-order derivative inspection or sign change check; e.g., x^3 has an inflection point at x = 0, whereas x^4 has a local minimum).

6. Worked Step-by-Step Examples

Worked Example 1: Limit with Indeterminate Power (1^∞)

Problem: Evaluate L = lim_{x → 0} (1 + 3x)^{2/x}.

Solution: This has the indeterminate form 1^∞.

Take the natural logarithm: ln L = lim_{x → 0} \frac{2}{x} ln(1 + 3x) = 2 \cdot lim_{x → 0} \frac{ln(1 + 3x)}{x}.

Applying L'Hôpital's rule (form 0/0): ln L = 2 \cdot lim_{x → 0} \frac{\frac{3}{1 + 3x}}{1} = 2 \cdot 3 = 6.

Exponentiating gives: L = e^6. (Alternatively, apply standard result lim_{u→0}(1 + u)^{1/u} = e).

Worked Example 2: Taylor Expansion & Truncation Error

Problem: Approximate f(x) = \sqrt{1 + x} up to the second degree around a = 0, and estimate \sqrt{1.2}.

Solution: Compute derivatives at 0:

f(0) = 1
f'(x) = \frac{1}{2}(1 + x)^{-1/2} ⇒ f'(0) = 1/2
f''(x) = -\frac{1}{4}(1 + x)^{-3/2} ⇒ f''(0) = -1/4

The second-order Taylor polynomial is: P_2(x) = 1 + \frac{1}{2}x - \frac{1}{8}x^2.

For x = 0.2: P_2(0.2) = 1 + 0.1 - \frac{0.04}{8} = 1 + 0.1 - 0.005 = 1.095.

Actual value: \sqrt{1.2} \approx 1.095445 (Relative error < 0.04%).

Worked Example 3: Global Optimization on a Closed Interval

Problem: Find the absolute minimum and maximum values of f(x) = 2x^3 - 3x^2 - 12x + 5 on the closed interval [-2, 4].

Solution:

1. Find interior critical points: f'(x) = 6x^2 - 6x - 12 = 6(x^2 - x - 2) = 6(x - 2)(x + 1) = 0 ⇒ x = -1, 2.

Both points lie within [-2, 4].

2. Evaluate f(x) at critical points and boundary endpoints:

  • f(-2) = 2(-8) - 3(4) - 12(-2) + 5 = -16 - 12 + 24 + 5 = 1
  • f(-1) = 2(-1) - 3(1) - 12(-1) + 5 = -2 - 3 + 12 + 5 = 12
  • f(2) = 2(8) - 3(4) - 12(2) + 5 = 16 - 12 - 24 + 5 = -15
  • f(4) = 2(64) - 3(16) - 12(4) + 5 = 128 - 48 - 48 + 5 = 37

Conclusion: The absolute maximum is 37 (at x = 4) and absolute minimum is -15 (at x = 2).

7. GATE Exam Traps & Common Distractors

  • Checking Endpoints: For optimization over a closed interval [a, b], never forget to evaluate the boundary points f(a) and f(b). Extrema frequently lie on the boundaries where f'(x) ≠ 0.
  • Local vs Global: Critical points with f''(x) > 0 only guarantee local minima. Global optimality requires either convexity over the domain or exhaustive comparison with boundary values.
  • L'Hôpital Preconditions: Do not apply L'Hôpital's rule unless you have explicitly verified that the expression is in indeterminate form (0/0 or ∞/∞). Differentiating a determinate limit leads to wrong answers.

8. Syllabus Practice & Topic Tracking

Test your understanding with exam-pattern practice problems:

Practice Calculus Questions or review the GATE DA Syllabus Index.