How do we tell the stable and unstable equilibrium mathematically? cuz using the definition in the text to determine what equilibrium i'm dealing with is ambiguous and obscure...
This is a very good question which is actually very general. Let's start with some definitions...
Equilibrium is a state of a system in which the variables which describe the system are not changing (note that a system can be in a dynamic equilibrium where things might be moving or changing, but some variable(s) which describe the system as a whole is(are) constant). One example you are all familiar with is a mechanical system in equilibium where positions of objects are not changing (ie. no net forces acting).
In a Stable equilibrium if a small perturbation away from equilibrium is applied, the system will return itself to the equilibrium state. A good example of this is a pendulum hanging straight down. If you nudge the pendulum slightly, it will experience a force back towards the equilibrium position. It may oscillate around the equilibrium position for a bit, but it will return to its equilibrium position.
In an Unstable equilibrium if a small perturbation away from equilibrium is applied, the system will move farther away from its equilibrium state. A good example of this is a pencil balanced on it's end. If you nudge the pencil slightly, it will experience a force moving it away from equilibrium. It will simply fall to lying flat on a surface.
Ok, now that we know more about equilibria it will be easier to determine what "kind" we have. Strictly speaking, mathematically we determine whether a mechanical equilibrium is stable or unstable by looking at the second derivative of the energy with respect to the coordinate of interest. If the equilibrium is at a minimum (second derivative is positive) the system is in a stable equilibrium. If the equilibrium is at a maximum (second derivative is negative) the system is in an unstable equilibrium.
However, there is a simpler way to quickly test whether an equilibrium is stable or unstable. If we think about the definitions, each involves the response to a small perturbation. So let's "apply" a perturbation. If we say change the position slightly, is there a net force? in what direction? does the perturbation result in a force which drives the system back toward equilibrium (stable) or away from equilibrium (unstable)?
If you can't easily picture the situation in your head (as with the pencil and the pendulum), what this means in practical terms is that you re-calculate the forces at some position near, but not at equilibrium and determine whether they are driving the system back towards equilibrium (again, this means stable) or away from equilibrium (this means unstable).
Hope that clarifies things!