Take-Home Messages

  • Back to learning objectives.

  • Ultimate goal of statistics is data analysis. E.g., PageRank, handwritten digit recognization, Netflix matrix completion, NNMF, ...

    Statistics is partly empirical and partly mathematical. It is now almost entirely computational.
    Kenneth Lange

Statisticians used to .. Now we spend all time ...

  • Two essential skills for modern statisticians: programming and computational algorithms.

    Data Scientist (n.): Person who is better at statistics than any software engineer and better at software engineering than any statistician.
    Josh Willis on Twitter

  • Numerical linear algebra

    • building blocks of most computing we do.
    • Use standard (and good) libraries (BLAS, LAPACK, ...) as much as possible!
    • Sparse linear algebra and iterative solvers such as conjugate gradient (CG) methods are critical for exploiting structure in big data.

  • Optimization

    • Convex programming (LS, LP, QP, GP, SOCP, SDP). Download and study Stephen Boyd's book, watch lecture videos by Stephen Boyd or take UCLA EE236B, EE236C by Vandenberghe, familiarize yourself with the good optimization softwares. Convex programming is becoming a technology, just like least squares (LS). Browse the documentation of cvx and Convex.jl to see which functions are implemented.

    • Generic nonlinear optimization tools: Newton, Gauss-Newton, quasi-Newton, (nonlinear) conjugate gradient, ...

    • Optimization tools developed by statisticians: Fisher scoring, EM, MM, ... Take UCLA Biomath 205 by Kenneth Lange for a thorough study of MM algorithms.

  • About coding:

    • 5 Pro-Tips For Data Scientistis To Write Good Code.

    • Perseverance and mentality. It's always YOU! Not computer, not software, not Windows, not roundoff error, ... We tend to blame everything else except ourselves and give up. For example, when implementing Newton typle algorithms, wrong derivative and/or Hessian calculation is the #1 source of problems. When implementing EM/MM algorithms, non-monotonicity always means bugs in our code or derivation. Bring the homework mentality into research.

    • Keeping updating a (computational) note along with code.

  • Enjoy (or hate) Julia?
    JuliaCon 2022 will be online (free registration): http://juliacon.org/2022/