Hi Scholars!
This last week I returned to a question:
What's "ill-conditioned" mean when optimizing?
Surprisingly, I recently found 3 ways to quantify
whether something is "well-behaved."
People may refer to
how the solution of a problem changes with input data,
the shape of the landscape of the objective function, or
the condition number of a matrix.
Although related, these are distinct concepts.
In the ideal "well-conditioned" problem,
small change in input 𝗱𝗮𝘁𝗮 → small change in 𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻;
small change in 𝗹𝗼𝗰𝗮𝘁𝗶𝗼𝗻 → small change in 𝗴𝗿𝗮𝗱𝗶𝗲𝗻𝘁;
𝘀𝗶𝗻𝗴𝘂𝗹𝗮𝗿 𝘃𝗮𝗹𝘂𝗲𝘀 of matrices (e.g. Hessians) are 𝗰𝗹𝗼𝘀𝗲.
The slides below walk through how to quantify this.
Enjoy!
Howard