Working Notes: a commonplace notebook for recording & exploring ideas.
Home. Site Map. Subscribe. More at expLog. — Kunal

Deep Learning Systems

To save some time, I'm going to speed read the slides / reading first, and then watch videos if it makes sense to me.

Videos:

• https://www.youtube.com/watch?v=ftP5HeOvsI0

Lecture 1

• mostly prep slides, and to check out the zeroth assignment

Lecture 2

• each model is basically

  • hypothesis with parameters mapping inputs to outputs
  • loss function specifying how well a hypothesis performs
  • optimization method for determining a set of parameters that approximately minimizes sum of losses on training set

• linear hypothesis function uses a linear operator == matmul for transformation

• classification error

  • simplest loss function
  • 0 if argmax(h(x)) == y, 1 otherwise

• softmax / cross entropy loss

  • z_i = p(label = i) = e(h(x)) / (sum e(h(x)))
  • loss to be the negative log probability of the true class = -log p (label=y) = -hy(x) + log(sum e(h(x)))

Lecture 3

• manual neural networks
• 2 layer network can approximate any function

Lecture 4

• numerical differentiation -- approximate the gradient,

  • not very efficient, and suffers from numerical errors

• Automatic Differentiation

• good way to check auto differentiation

  • differentiate in reverse topological order

• partial adjoints for handling multiple pathways

• notes/neural_nets
• notes/gradient_descent
• notes/deep_learning_nyu/deep_learning_nyu
• notes/deep_learning_nyu/deep_learning_nyu_week_2
• notes/softmax
• notes/deep_learning_nyu/deep_learning_nyu_week_5
• notes/pattern_recognition_&_machine_learning
• notes/deep_learning_nyu/deep_learning_nyu_week_1
• notes/loss_function
• notes/deep_learning_nyu/deep_learning_nyu_week_3