How to group ports by physical adapters in ESXi using pyvmomi ?

Recently I needed a way to find the number of physical adapters and also find the ports in the physical adapters in ESXi using pyvmomi.

I googled for examples. However, I did not find a direct answer. So, in this post, I would like to show the code I used to achieve this.

The logic I used is: Ports of a physical adapter will have the same first 5 bytes of MAC or the same device id or bus+slot combination

class VMWareClient:
    def __init__(self, host, user, password, port=443, insecure=True):
       if insecure:
           self._service_instance = SmartConnectNoSSL(host=host, user=user, pwd=password, port=port)
       else:
           self._service_instance = SmartConnect(host=host, user=user, pwd=password, port=port)
       self._host = host
       self._user = user
       self._password = password
       atexit.register(Disconnect, self._service_instance)
       self._content = self._service_instance.RetrieveContent()

    def get_pnics(self):
        host_view = self._content.viewManager.CreateContainerView(self._content.rootFolder, [vim.HostSystem], True)
        hosts = [host for host in host_view.view]
        host_view.Destroy() 
        host = hosts[0]
        pnics = []
        for pnic in host.config.network.pnic:
            pnic_details = {}
            pnic_device = pnic.device
            if pnic_device.startswith('vmnic'):
                if pnic.pci:
                    for pci_device in host.hardware.pciDevice:
                        if pci_device.id == pnic.pci:
                            vendor = pci_device.vendorName
                            device = pci_device.deviceName
                            driver = pnic.driver
                            mac = pnic.mac
                            device_id = pci_device.deviceId 
                            if pnic.linkSpeed:
                                status = 'Connected'
                            else:
                                status = 'Disconnected'
                            pnic_details['name'] = pnic_device
                            pnic_details['vendor'] = vendor
                            pnic_details['device'] = device
                            pnic_details['driver'] = driver
                            pnic_details['status'] = status
                            pnic_details['mac'] = mac
                            pnic_details['device_id'] = device_id 
                            pnics.append(pnic_details)
        return pnics

def print_adapters_ports(self):
    pnics = self.get_pnics()
    print('Grouping ports by adapters based on MAC')
    pnics_by_mac = sorted(pnics, key=lambda x:x['mac'][:-3])
    for key, value in itertools.groupby(pnics_by_mac, key=lambda x:x['mac'][:-3]):
        print(key)
        for i in value:
            print(i.get('name'), i.get('mac'))
        print('\n')

    print('Grouping ports by adapters based on device id')
    pnics_by_device_id = sorted(pnics, key=lambda x:x['device_id'])
    for key, value in itertools.groupby(pnics_by_device_id, key=lambda x:x['device_id']):
        print(key)
        for i in value:
            print(i.get('name'), i.get('mac'))
        print('\n')

    print('Grouping ports by adapters based on bus+slot')
    pnics_by_bus_slot = sorted(pnics, key=lambda x:x['id'][5:10])
    for key, value in itertools.groupby(pnics_by_bus_slot, key=lambda x:x['id'][5:10]):
        print(key)
        for i in value:
            print(i.get('name'), i.get('mac'))
        print('\n')

 

Advertisements

How to implement a simple logging server using Python gevent ?

Recently, I started working on a tool which required me to implement a logging service which can write the input logs to a file.

I tried different libraries. However, using gevent made my life simpler.

Here is the code for the logging server using gevent.

Call the below file as in below example

Recently, I started working on a tool which required me to implement a logging service which can write the input logs to a file.

I tried different libraries. However, using gevent made my life simpler.

Here is the code for the logging server using gevent.

#gevent_logging_server.py

import sys

from gevent.server import StreamServer


def write_logs_to_file(socket, address):
    rfileobj = socket.makefile(mode='rb')
    unique_file_name = 'log_file_unique'
    log_file = open("{}/{}.log".format(logging_server_dir, unique_file_name), 'ab')
    while True:
        line = rfileobj.readline()
        if not line:
            msg = b"client disconnected"
            log_file.write(msg)
            break
        if line.strip().lower() == b'quit':
            msg = b"client quit"
            log_file.write(msg)
            break
        log_file.write(line)
    rfileobj.close()
    log_file.close()


if __name__ == '__main__':
    logging_server_ip = sys.argv[1]
    logging_server_port = int(sys.argv[2])
    logging_server_dir = sys.argv[3]
    server = StreamServer((logging_server_ip, logging_server_port), write_logs_to_file)
    server.serve_forever()

Call the above file as in below example

python gevent_logging_server.py 192.168.10.20 5000 /home/logging_server/logs

How to create a custom ISO image from RHEL 7 ISO using genisoimage ?

Recently I had to delete few packages in RHEL 7.6 ISO.

I tried few softwares in Windows like 7zip, gburner, anyburn, etc

However, none of them helped me.

So, one of my colleagues suggested me to use mkisofs.

I never used it before. So, I had to spend some time googling on how to use mkisofs.

Below is the list of commands I ran in Ubuntu 16.04 which helped me create a custom ISO from RHEL 7.6 ISO.

My ISO name is RHEL-7.6-Server-x86_64-dvd1.iso

  1. Create few directories
mkdir /home/play_with_iso
mkdir /home/play_with_iso/original
mkdir /home/play_with_iso/mnt
mkdir /home/play_with_iso/copy_iso_files
mkdir /home/play_with_iso/new

     2. Copy the ISO to /home/play_with_iso/original

     3. Mount the ISO

mount -t iso9660 -o loop /home/play_with_iso/original/RHEL-7.6-Server-x86_64-dvd1.iso /home/play_with_iso/mnt

     4. Copy the contents of the mounted ISO

cp -pRf /home/play_with_iso/mnt /home/play_with_iso/copy_iso_files

   5. Unmount the ISO

umount /home/play_with_iso/mnt

   6. Modify the contents of the ISO in the path /home/play_with_iso/copy_iso_files

   7. Create new ISO.

cd /home/play_with_iso/copy_iso_files
genisoimage -U -r -v -T -J -joliet-long -V "RHEL-7.6 Server.x86_64" -volset "RHEL-7.6 Server.x86_64" -A "RHEL-7.6 Server.x86_64" -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -eltorito-alt-boot -e images/efiboot.img -no-emul-boot -o /home/play_with_iso/new/RHEL-7.6-Server-x86_64-dvd1.iso .

Note:

The value of the parameters V, volset, A should be the value of inst.stage2 in isolinux.cfg .

In the ISO I used, the value is RHEL-7.6\x20Server.x86_64

Look for a line in isolinux.cfg as below

append initrd=initrd.img inst.stage2=hd:LABEL=RHEL-7.6\x20Server.x86_64 quiet

So, the values of the parameters V, volset, A should be “RHEL-7.6 Server.x86_64”

  8. Implant an MD5 checksum into the image

implantisomd5 /home/play_with_iso/new/RHEL-7.6-Server-x86_64-dvd1.iso

 

Reference Links

  1. RHEL 7 Working with ISO Images

How to set default templates to an OS in Foreman 1.15.6 using REST API ?

I recently started using Foreman for provisioning servers. It is a nice tool, however, the documentation is not so great.


I did not find the REST API documentation for setting default templates to an OS. 

So, I ran the hammer set-default-template command with verbose option and got the REST API I needed.

Below are the simple functions which can help set default templates to an OS

import requests
requests.packages.urllib3.disable_warnings()


FOREMAN_URL = 'https://192.168.10.20/api'
FOREMAN_CREDENTIALS = ('admin', 'foreman')

HEADERS = {'content-type': 'application/json'}


def get_template_by_id(template_id):
    url = '{}/{}/{}'.format(FOREMAN_URL, "config_templates", template_id)
    template = requests.get(url, auth=FOREMAN_CREDENTIALS, verify=False)
    return template.json()

def set_os_default_templates(os_id, template_ids):
    for template_id in template_ids:
        template = get_template_by_id(template_id)
        url = '{}/{}/{}/{}'.format(FOREMAN_URL, "operatingsystems", os_id, "os_default_templates")
        payload = {"os_default_template": {"config_template_id": str(template_id), "template_kind_name":template['template_kind_name']}}
        requests.post(url, headers=HEADERS, auth=FOREMAN_CREDENTIALS, json=payload, verify=False)

# Set templates with ids 10, 20 as default templates to OS with id 1        
os_id = 1
template_ids = (10, 20)
set_os_default_templates(os_id, template_ids)

How to install and configure samba in Ubuntu 18.04 ?

I recently started using samba to create network shares for storing ISO images.

Below are the commands which helped me setup samba in Ubuntu 18.04

  1. Install samba
apt-get install samba

2. Create a user and password for accessing samba share

useradd sambauser --shell /bin/false
smbpasswd -a sambauser

3. Create a directory for storing ISO images and change ownership

mkdir /home/sambauser/iso_images
chown sambauser /home/sambauser/iso_images

4. Edit samba configuration. Add the below content to the end of the file /etc/samba/smb.conf

[iso_images]
path = /home/sambauser/iso_images
valid users = sambauser
read only = no

5. Restart samba

   service smbd restart

6. Verify samba. If the IP of the above samba host is 192.168.10.20, we can verify the share using the below instructions

Windows
Open Run and type \\192.168.10.20\iso_images

Ubuntu
apt-get install cifs-utils
mount -t cifs //192.168.10.20/iso_images /home/iso_images -o rw

Create few files in the samba directory and check if they are available in the shared location

 

How to send vm-support logs in ESXi via kickstart file ?

I recently started using ESXi and I need to install ESXi via kickstart files.

The basic kickstart file available in different websites works fine.

However, I needed a way to send vm-support logs to a FTP server.

Assuming FTP server IP is 192.168.10.20 and the directory “esxi_vmsupport_logs” exists in FTP root directory, below is the post section of the kickstart file

 

%post --interpreter=busybox --ignorefailure=true

#disable firewall to send logs to FTP server
localcli network firewall set --enabled=false

vm-support -s > /tmp/vm-support-logs.tgz

cat >/tmp/send_vmsupport_logs.py << EOF
from ftplib import FTP
import subprocess
import sys


def send_logs_to_ftp(ftp_ip, ftp_dir, ftp_user='ftp'):
    ftp = FTP(ftp_ip)
    ftp.login(ftp_user)
    ftp.cwd(ftp_dir)
    service_tag = subprocess.check_output("localcli hardware platform get | sed -n '/^\s*Serial/p' | cut -d: -f 2 | cut -d' ' -f 2", shell=True).strip()
    with open('/tmp/vm-support-logs.tgz', 'rb') as f:
        dest_log = service_tag.decode("utf-8") + '.tgz'
        ftp.storbinary('STOR %s' % dest_log, f)
    ftp.quit()


if __name__ == '__main__':
    ftp_ip = sys.argv[1]
    ftp_dir = sys.argv[2]
    send_logs_to_ftp(ftp_ip, ftp_dir)
EOF
python /tmp/send_vmsupport_logs.py 192.168.10.20 /esxi_vmsupport_logs

reboot

 

How to setup anonymous FTP and transfer files using Python ftplib ?

I recently started using vsftpd and Python ftplib in one of my projects to save log files in FTP.

I would like to list down the instructions, configuration and sample code which helped me setup anonymous FTP and upload files using Python ftplib

Note: My host OS is Ubuntu 18.04

Install vsftpd

apt-get update && apt-get install vsftpd

Create directory for FTP logs

mkdir /home/ftp_logs

Make the directory world writable

chmod a+rwx /home/ftp_logs

Set ownership for the directory

chown -R nobody:nogroup /home/ftp_logs

Configure vsftpd (/etc/vsftpd.conf)

listen=YES
local_enable=NO
anonymous_enable=YES
write_enable=YES
anon_root=/home/ftp_logs
allow_writeable_chroot=YES
virtual_use_local_privs=YES
anon_world_readable_only=NO
anon_upload_enable=YES
anon_mkdir_write_enable=YES
no_anon_password=YES

Start vsftpd

/etc/init.d/vsftpd start

Python code to transfer a file using ftplib

from ftplib import FTP

FTP_IP = '192.168.10.20'
ftp = FTP(FTP_IP)
ftp.login('ftp')

with open('/home/install_status_src.log', 'rb') as f:
    ftp.storbinary('STOR %s' % 'install_status_dest.log', f)
ftp.quit()

Run this code to upload a file to FTP server.