Using Python To Prevent A System Crash

An issue with a critical server had been plaguing the IT department for upwards of a year without any resolution.

The issue stemmed from the server running out of virtual memory thus resulting in a crash.  This would happen every month or so.  The IT staff on board at the time were unable to find the cause of the issue and resorted to rebooting the server during the monthly maintenance window as a fix.

The server was running CentOS 6.3 and was being monitored using Nagios.

Shortly after I came aboard I found the following in the syslog file (/var/log/messages) after a crash by cross referencing the time Nagios reported the server being unavailable:

Sep 27 03:34:47 sit-admin kernel: smbd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
Sep 27 04:03:55 sit-admin kernel: smbd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
Sep 27 04:03:55 sit-admin kernel: smbd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0

The aforementioned error was a result of a memory leak in the version of Samba running on the server.  The error stems from the smbd daemon creating a new process every time a connection is attempted, successful or not.

The Linux kernel allows processes to request virtual memory whether there is enough memory available or not.  The kernel has a function called Out-Of-Memory Killer (oom-killer) that will kill processes that are of lower priority in order to keep the system running.

After many connections there were over 100 orphaned smbd processes on the system, thus exhausting memory and crashing the system.

Do to some (mis)configuration issues on a another person's part, we were unable to update the version of Samba in a straight forward manner. 

In order to circumvent further crashes I wrote a Python script that checks the for the amount of swap space available will restart the Samba daemon (or whatever daemon you specify) if the amount below the specified value.

The script can be found here.  I scheduled the script to run as cron job and no further crashes have occurred.

Create User Accounts From A csv File In Linux Using Python

Here's a script that I wrote that creates user accounts and home directories from a csv in Linux using Python.

This is a common system administration task that be done using shell scripts but is much easier using Python.

Python has a handy csv module that parses csv files so you do not have to worry about it yourself.

I plan to focus my studies on Linux after I complete the MCITP: Enterprise Messaging Administrator 2010 certification.  So there will be more scripts to come.

#!/usr/bin/python
#pyuseradd.py
#Read users from a csv file and create accounts and home directories.
#Sheldon Alman - sheldonalman@gmail.com
#csv file format : firstname, lastname, username, password
import sys, csv, subprocess

if len(sys.argv) != 2:
    print "Usage: " + str(sys.argv[0]) + " filename"
else:
    filename = str(sys.argv[1])
    with open(filename,  'rb') as csvfile:
        accounts = csv.reader(csvfile)
        for row in accounts:
            subprocess.call(['useradd', '-m' , '-s' + '/bin/bash','-c' + row[0] + row[1] ,  row[2]])
            subprocess.call('echo ' + row[2] +":" + row[3] + " | " + "chpasswd",  shell=True )

Some More Python & Nmap

As an exercise, I decided to write a Python script that performed an Nmap scan on my home network and showed the difference between scans.

I found that Nmap has the option to output to XML format and that a style-sheet is included with Nmap to allow for easier viewing.

ndiff, which, comes with Nmap is used to show the differences between Nmap scans like the Unix/Linux diff command does with files.  It has the ability to output to XML format, but a style-sheet isn't included and Nmap's style-sheet doesn't format ndiff's XML output correctly.

yandiff is an improvement to ndiff that adds more functionality, including a XML style-sheet.

The script I wrote is a little rough, but it works in the following way;
1) Run nmap with the "-A" and "-PN" arguments against the specified network and output to XML format with the date timestamped in the file name.
2) Use yandiff to compare the current week's scan with the previous week's scan and output the result to XML format with the date timestamped in the file name.
3) Use xsltproc to convert the Nmap scan results and the yandiff results to HTML

I then scheduled a cron job to run the script weekly.

Update:
I wrote another version of the script that uses os.fork() and the os.exec*() family of functions.  I did this to make the script a little cleaner and also to learn about process management.  The results are the same as the previous script, just done a little differently.

I learned an interesting lesson on how global variables and forking to create new processes work.

The code can be found here.

Below are samples of the results:

Some More Python

As I work through the exercises from Core Python Programming 2e I'll post my solutions to some of the exercises that I find interesting.

Here's a wrapper for the Windows command-line interface that allows one to use the Unix/Linux ls, more, cat, cp, mv and rm commands to manipulate files in Windows:

import os
import string

class shell(object):
    def cmd(self):
        while 1:
            cmdLine = raw_input("$ ")
            if "ls" in cmdLine and len(cmdLine) == 2:
                os.system("dir")
            if "ls" in cmdLine[:cmdLine.find(" ")]:
                path = cmdLine[cmdLine.find(" "):]
                os.system("dir" + path)
            if "more" in cmdLine[:cmdLine.find(" ")]:
                path = cmdLine[cmdLine.find(" "):]
                os.system("more" + path)
            if "cat" in cmdLine[:cmdLine.find(" ")]:
                path = cmdLine[cmdLine.find(" "):]
                os.system("type" + path)
            if "cp" in cmdLine[:cmdLine.find(" ")]:
                path1 = cmdLine[cmdLine.find(" "):cmdLine.rfind(" ")]
                path2 = cmdLine[cmdLine.rfind(" "):]
                os.system("copy" + path1 + path2)
            if "mv" in cmdLine[:cmdLine.find(" ")]:
                path1 = cmdLine[cmdLine.find(" "):cmdLine.rfind(" ")]
                path2 = cmdLine[cmdLine.rfind(" "):]
                os.system("ren" + path1 + path2)
            if "rm" in cmdLine[:cmdLine.find(" ")]:
                path = cmdLine[cmdLine.find(" "):]
                os.system("del" + path)

Data Structures in Python

Currently I'm working on learning Python and have been working through various books.

A few of the exercises in the book Core Python Programming 2e ask the reader to implement data structures as classes, specifically stacks and queues.

For those who do not know what a data structure is, you can read about it here.  A stack is a Last-In-First-Out (LIFO) data structure and a queue is a First-In-First-Out (FIFO) data structure.

I chose to use Python's built in list functionality to implement the classes.  The solutions proved to be very simple as opposed to the same solutions being produced in a language like C.

Here is the code:

Stack

class stack(object):
    def __init__(self):
        self.stack = list()

    def push(self, item):
        self.stack.append(item)

    def pop(self):
        self.stack.pop(len(self.stack)-1)

    def isempty(self):
        if len(self.stack) == 0:
            return True
        else:
            return False
   
    #displays the last element of the stack
    def peek(self):
        return self.stack[len(self.stack)-1]

Queue

class queue(object):
    def __init__(self):
        self.queue = list()

    def enqueue(self, item):
        self.queue.append(item)

    def dequeue(self):
        return self.queue.pop(0)

    def isempty(self):
        if len(self.queue) == 0:
            return True
        else:
            return False

    #displays the first element of the queue
    def peek(self):
        return self.queue[0]

Hybrid Stack/Queue

class stackQueue(object):
    def __init__(self):
        self.stackQueue = list()

    def shift(self):
        return self.stackQueue.pop(0)

    def unshift(self, item):
        self.stackQueue.insert(0,item)

    def push(self, item):
        self.stackQueue.append(item)

    def pop(self):
        return self.stackQueue.pop(len(self.stackQueue)-1)

    def isempty(self):
        if len(self.stackQueue) == 0:
            return True
        else:
            return False

     #returns the first element of the data structure
    def peek(self):
        return self.stackQueue[0]

First Solo Python Script

As a follow up to the last post, I've been learning Python from various books and have been soley following the exercises in the books themselves and not venturing out on my own until now.

I decided to try writing a simple script that installs nmap 6.01 from source on a Unix/Linux platform.  You can find it here.  The script was just an exercise to see if I could actually get it working.  I may make additions/modifications to it later on.

Learning Python

Over the past year or so I've been learning Python.  I have a Computer Science degree but strayed away from programming.  However, after developing an interest in IT Security I found that it is very beneficial to know how to at least read code let alone write it.

Python is very popular in the Security field and is gaining popularity in the Unix/Linux systems administration field to the point where it is directly competing with Perl.

I've been learning from the following books; Python Programming: An Introduction to Computer Science, Practical Programming: An Introduction to Computer Science Using Python, and Core Python Programming.

In terms of self-study.  If you have no background in programming or Computer Science I would recommend starting with Python Programming and moving on to Core Python.  If you have a programming/Computer Science background you can start with Core Python.

Python Programming teaches Computer Science using Python as the language.  Core Python assumes one already has programming experience.  Core Python focuses on the language and then applications using the language.

As for Practical Programming.  I found this title to be very poor for self-study and the authors seem to make many assumptions to the point where they just throw code at you without thorough explanation.