Hi Scholars!

This last week I returned to a question:

What's "ill-conditioned" mean when optimizing?

Surprisingly, I recently found 3 ways to quantify

whether something is "well-behaved."

People may refer to

how the solution of a problem changes with input data,

the shape of the landscape of the objective function, or

the condition number of a matrix.

Although related, these are distinct concepts.

In the ideal "well-conditioned" problem,

small change in input 𝗱𝗮𝘁𝗮 → small change in 𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻;

small change in 𝗹𝗼𝗰𝗮𝘁𝗶𝗼𝗻 → small change in 𝗴𝗿𝗮𝗱𝗶𝗲𝗻𝘁;

𝘀𝗶𝗻𝗴𝘂𝗹𝗮𝗿 𝘃𝗮𝗹𝘂𝗲𝘀 of matrices (e.g. Hessians) are 𝗰𝗹𝗼𝘀𝗲.

The slides below walk through how to quantify this.

Enjoy!

Howard