Monitoring
Nagios has become an industry standard for open source monitoring. Several projects such as Groundworks use Nagios as their core engine; but our experience is that Nagios 3.x is still the most reliable and flexible solution.
We have deployed the community edition of Xenoss, called Xenoss Core, which allows for easy configuration of both monitoring and graphing. Plugins can be added to pull extended statistics from applications like Apache, Squid, and MySQL.
If only graphing and trending is required, Cacti is a great solution: it uses the powerful RRDtool. If you need assistance with setting up some other package, we can help with that too.
There are many small command line tools that are vital for network debugging, performance (I/O) tracking, and process debugging. Knowing where to look is the key to debugging many wierd issues on Linux.
If nothing existing provides the information we need to solve a problem, Perl or BASH can often be used to pull it together.