Sys Admin Docs

• Monitoring
  • problems in dayabay context
  • nagios
  • others
  • fabric-cuisine-watchdog/daemonwatch
• Mars
• Migration from Trac/SVN
  • Conversion of Trac wiki to Sphinx rst
    • Status of TracWiki2Sphinx
      • establish development cycle without the webserver
    • Strategy
    • Trac Formatter based
    • Metadata preservation
    • Sphinx access to docinfo metadata
    • Trac Macros that I use
      • TracNav(ReponameNav) : sidebar navigation panel
      • PageOutline
      • TagCloud
      • TagList + per-page tag listing
  • Trac2Github
    • SVN to git
      • recipe for git to svn conversion
      • git-svn
      • subgit : keeps both alive
    • Trac issues to Github issues
    • Trac wiki content to Sphinx content
    • XMLRPC access to Trac
• SOP : for servers
  • general
  • hfag
    • hfag restart requires manual F2 keypress
    • hfag nginx reverse proxy
  • cms02 : as repo server this one goes last
  • cms02 : restarting scm backup system after a reboot
    • check/remove LOCKED backup dirs
    • restore ssh agent
    • check/edit crontab times
  • cms01
    • heprez servers : exist httpd tomcat
    • supervisord and contained mysql
    • rabbitmq-server
    • xinetd
• cms01
  • Jul 30, 2015 Manual Stop prior to powercut
  • Dec 26, 2014
  • Oct 23, 2014
  • cms01 /data mount
  • jun 29 2014, unusual disk usage bump : EXPLAINED
• CMS02
  • July 30, 2015 Manual Stop prior to powercut
  • Oct 14, 2014
  • Sep 4, 2014
  • Sep 1, 2014
  • Aug 29, 2014
  • Aug 4, 2014
  • Following Typhoon Matmo C2R to H1 backups failing
  • Attack 19/Jun/2014 from 183.60.119.35
  • Confirmed Robot Attack 20/Jun/2014 from 58.254.168.39
  • Normal Hourly Hits
  • Jun 20, 2014 : again
  • Jun 19, 2014 : httpd offline, OOM again
    • hourly valmon monitoring fails from 06:42
    • Original Cause, httpd OOM
    • Restart httpd
• HFAG
  • Manual Take Down July 30th, 2015
• Belle7
  • Block from simon
  • traceroute not helping
• cms01 DNS HTTP 10 seconds delay
  • cms01 access monitoring is failing
  • Other Observations
  • DNS /etc/resolv.conf
  • Fixed with dns-edit
• Network Troubleshooting
  • Overview
  • RESOLVED as due to accidental miscabling
  • Sequence of events
  • Network Troubleshooting Refs
  • pinging
  • Troubleshooting Tools
    • ifconfig
      • ethtool
    • mii-tool
    • arp
    • dmesg
    • netstat
  • Checking configuration
  • Stop/start network service
  • cms02 inet6 redherring
  • Checklist
  • Check Config
• HGPU01
  • Quick Access
  • Access
  • Mercurial
  • Network
  • Storage
  • Headless OpenGL ?
  • libPNG