Thursday, 25 June 2015

Set CPU family for a specific VM on oVirt 3.5

I have been playing around with an oVirt test lab now on and off for a couple of months and I must say I love it. The community is active and helpful (either via the users@ovirt.org mailing list or #ovirt on irc.oftc.net using IRC) and the software is continuously upgraded and improved by a well organised development team.

One of the features that is missing in version 3.5 is the ability to set the CPU family on a VM to CPU family with a lower set of features than the current hardware (actually the cluster but that is likely to be set to the same thing). Sounds daft, why would anyone want to do this, well you can test if a VM would work on a older class of hardware I suppose. What I wanted it for a bug with Windows 10 previews that throw a SYSTEM THREAD EXCEPTION NOT HANDLED which prevents installing on a SandyBridge CPU on the KVM hypervisor (the hypervisor used by oVirt). The answer is to fall back to the older CPU family of Westmere. To do that on stock 3.5 oVirt you need to set the whole cluster to that CPU family, there is not option to do just a VM.

I presumed someone else had figured a way to do this so asked on the #ovirt IRC channel, no one knew but did try to put me in touch with some who might help but they were away. I then emailed the mailing list and got back two prompt answers.  First saying I need to use something called VDSM hooks and another says the feature will be available in 3.6 if I can wait or upgrade before final release.

I looked up VDSM hooks, these are (mostly python) scripts that can effect certain events on a running ovirt engine. The event I needed to look at is before_vm_start, following instructions and modifying the example script on this page I was able to customise my lab to enable CPU setting.
Create the following script on each host in the cluster.
vi /usr/libexec/vdsm/hooks/before_vm_start/50_cpufamily
Paste in the following text:
#!/usr/bin/python

import os
import sys
import hooking
import traceback


if os.environ.has_key('cpufamily'):
    try:
       domxml = hooking.read_domxml()   #here we read the VM XML into the domxml variable
       vcpu = domxml.getElementsByTagName('cpu')[0] #find and read the CPU definition in the VM XML
       sys.stderr.write('cpufamily: Changing cpu family to: %s\n' % os.environ['cpufamily'])  #sys.stderr.write is caught by vdsm and logged into vdsm.log for debugging
       e = domxml.createElement('model')
       txt = domxml.createTextNode(os.environ['cpufamily'])
       e.appendChild(txt)
       modelnode = vcpu.getElementsByTagName('model')[0]
       vcpu.replaceChild(e,modelnode)
       hooking.write_domxml(domxml)                       #and write to the altered domxml
    except:
       sys.stderr.write('cpufamily: [unexpected error]: %s\n' % traceback.format_exc())
       sys.exit(2)

Change the script to be executable:
# chmod +x /usr/libexec/vdsm/hooks/before_vm_start/50_cpufamily
If you are using the hosted engine setup on one of the hosts put the HA agents into global maintenance with:
# hosted-engine --set-maintenance --mode=global
Then on just the engine machine:
# engine-config -s UserDefinedVMProperties='cpufamily=^(Conroe|Penryn|Nehalem|Westmere|SandyBridge)$' --cver=3.5
# service ovirt-engine restart
Take the ha agents out of maintenance mode:
# hosted-engine --set-maintenance --mode=none
This should now allow you to log into the engine GUI, edit a VM (or create a new one) under advanced options go to custom properties and select the cpufamily key and set an appropriate CPU family, the next time you boot your VM will only have access to those CPU features relevant to the CPU family you have selected.

Monday, 16 February 2015

Genius open-vm-tools

I have always felt that running a Debian VM in a VMware environment was a little bit of a pain, installing the vmware tools was easy enough that you could do it in a few minutes but not so easy that I could remember how to do it (hence this post).

Now VMware fully support open-vm-tools things are much easier if you are running wheezy or newer, you can simply run:

apt-get install open-vm-tools

Then everything just works.

I look forward to the day when the Debian installer detects that it is getting installed on a VMware VM and does this automatically for you at install.

Wednesday, 5 November 2014

Powershell script to bring up Hyper-V VMs Slowly enough that they function

The below Powershell script is something I wrote out of frustration while on an Exchange Messaging course to try to improve my mood. The Hyper-V VMs we were starting for each lab took an absolute age to load up and become responsive after we reverted their state at the end of each lab. This script attempts to start machines one after another with a configurable gap in between, the idea being you could set the $vms array to the machines you want started then wander off for a bit of a break then hopefully when you return the VMs would have loaded and be responsive.

You may ask why wouldn't you just start them all together, well the machines that were running the VMs were not up to spec and each of the VMs seemed to be based off of a single VM using dereferenced disks which severely impacted performance. The VMs took so long to load that Services on each of them often failed to start because of timeouts.

I thought I would share in the hope it may save someone else pulling their hair out.

# Machines to start in order to start
$vms = "20342B-LON-DC1", "20342B-LON-CAS1", "20342B-LON-MBX1", "20342B-LON-CL1", "20342B-LON-CL2" , "20342B-LON-LY1"
# minimum uptime in seconds to check for before starting next machine
$delaytime = 45
Function StartVM
 {
     param ($Name)
     $vmquery = get-vm $Name
     if ($vmquery.state -eq "off")
     {
        Write-Host " not running, starting" -NoNewline
        start-vm $Name
        $vmquery = get-vm $Name
     }
     elseif ($vmquery.state -eq "Running")
     {
        Write-Host " already running" -NoNewline
     }
     While ($vmquery.uptime.TotalSeconds -lt  $delaytime) { $vmquery = get-vm $Name; sleep 2 ; Write-Host . -NoNewline }  
     write-host " uptime is "-nonewline
     write-host $vmquery.uptime.TotalSeconds -NoNewline
     write-host " seconds, over $delaytime so assumed to be up and running."
}
foreach ($vm in $vms) {
    write-host "Checking $vm... " -NoNewline
    StartVM $vm 
}
write-host "If $delaytime seconds was long enough your VMs should be functioning now."


I am still on the course studying for 70-341: Core solutions of Microsoft Exchange Server 2013 and 70-342: Advanced Solutions of Microsoft Exchange Server 2013, So I may well update the script as I run more labs. Ideally I would like to improve the script so it waits until the started VM is "responding" before starting the next one but this has proved difficult, if you have any suggestions please let me know.

Update: 06/02/2015, I have improved the script a little, it now copes better with machines being started from other sources or if the script is restarted and the times are in seconds as the latest tests I have endured have been on lab machines using SSDs which significantly improves performance.

Tuesday, 9 September 2014

Squeezing more from your Linux VMs

There is a way to squeeze a little more from Linux VM's.

The theory

Linux has a number of ways of sharing it's storage IO among different processes, the norm seems to be to use the Completely Fair Queuing (CFQ) scheduler which helps to prevent a single process from using more than it's fair share of storage IO. This is usually helpful in an environment where this one Linux box is the only OS using a storage device but when it is running on an hypervisor the hypervisor is also busy trying to dish out fair access to the storage so we are effectively doubling up calculating fair access. There are other schedulers that we can choose from and for VM's it seems to make the most sense to use either the noop or the deadline schedulers. They are both more simple to calculate then CFQ.

The practice

If you are using the device sda for your drive use the following to check what scheduler you are using :
# cat /sys/block/sda/queue/scheduler
which will return something like
noop anticipatory deadline [cfq]
Which indicates that the scheduler for this device is CFQ.

To change it on the fly you just: 
echo noop > /sys/block/sda/queue/scheduler
If you want to make the change work across reboots you will need to add elevator=noop to your kernel boot parameters, on a Debian system you edit /etc/default/grub and add "elevator=noop" to the GRUB_CMDLINE_LINUX line then run:
# update-grub
To update the grub configuration.

Sources: kb.vmware.com/kb/2011861http://serverfault.com/questions/360718/kvm-low-io-performance

Sunday, 15 December 2013

Remote desktop gateway connections failing when KB2592687 installed.

While setting up a lab running Remote Desktop Gateway on Windows Server 2012 R2 I came across a strange problem. I was able to connect through the RD Gateway using some machines but not others.
On the machines that were failing I got the following message when trying to connect:
"Your computer can't connect to the remote computer because an error occurred on the remote computer that you want to connect to. Contact your network administrator for assistance."

After much pulling of hair I narrowed it down to something to do with the KB2592687 update. This update installs RDP version 8.0 on Windows 7 SP1 machines. This update was present on both the machine I was trying to connect to and on the clients that were failing. There is a list of known problems in knowledge base article but none of them applied to my setup.

More searching found loads of red herrings, then I discovered this post. It mentions LAN manager authentication level settings (Local security policy->Local Policies->Security Options->Network Security: LAN Manager authentication level). On the failing client it was set to "Send LM & NTLM - user NTLMv2 session security if negotiated." Changing it to "Send NTLMv2 response only" (which seems is the default on Vista and above) made the connection work.


Sunday, 16 June 2013

Extending disks on a Windows VM

Extend disks on a Windows Server 2008 R2 machine (and probably Windows 7 as well).

Extend the hard drive in VM Settings, this can be done while Windows is running but I was unable to get Windows to see the new size until after I rebooted the VM.

GUI
Server manager->Storage->Disk management, right click on the drive and choose "Extend Volume..." select size to expand to then click next.

Command line way
Start->Run and type diskpart.
At the disk part prompt type:
DISKPART> list disk

  Disk ###  Status         Size     Free     Dyn  Gpt
  --------  -------------  -------  -------  ---  ---
  Disk 0    Online           40 GB  5121 MB

This will show you a list of the disks in the system, once you have identified the disk you want by it's number (I only have disk 0 (zero) in the example above).
We now need the volume number:

DISKPART> list volume
  Volume ###  Ltr  Label        Fs     Type        Size     Status     Info
  ----------  ---  -----------  -----  ----------  -------  ---------  --------
  Volume 0     D   CD           CDFS   DVD-ROM      112 MB  Healthy    
  Volume 1         System Rese  NTFS   Partition    101 MB  Healthy    System
  Volume 2     C                NTFS   Partition     40 GB  Healthy    Boot
Now "select" the volume you want to extend (in this case I am extending my C drive so it is volume 2).

DISKPART> select volume 2

Volume 2 is the selected volume.

Now tell diskpart to extend the volume into the all the free space on disk 0:


DISKPART> extend disk=0

DiskPart successfully extended the volume.
That is it, all done.


Another easy way to extend disk sizes (as well as many other useful functions) is to run a live CD of gparted. This will also works with other operating systems such as Linux. In fact it should work any file system that gparted understands, check out there features page.

Monday, 13 May 2013

Setting up a branch office VPN between a Watchguard XTM505 and a Debian linux box.

Lets us say you have two sites and want to build a VPN between them. The first site has a Watchguard XTM device has a public IP address of 99.99.99.99 which protects a private IP address range of 10.0.1.0/24 and your second site has just a Debian Linux box which has a dynamic IP address that protects a private IP address range of 10.0.2.0/24


Debian side

Install needed packages:


# apt-get install ipsec-tools racoon
            Choose "direct" for racoon setup

Edit /etc/ipsec-tools.conf

#!/usr/sbin/setkey -f

# NOTE: Do not use this file if you use racoon with racoon-tool
# utility. racoon-tool will setup SAs and SPDs automatically using
# /etc/racoon/racoon-tool.conf configuration.
#

## Flush the SAD and SPD
#
# flush;
# spdflush;

spdadd 10.0.1.0/24 10.0.2.0/24 any -P in ipsec
           esp/tunnel/99.99.99.99-10.0.2.254/require;
spdadd 10.0.2.0/24 10.0.1.0/24 any -P out ipsec
           esp/tunnel/10.0.2.254-99.99.99.99/require;

Edit /etc/racoon/psk.txt and add the line

99.99.99.99   Somerandomkey

Edit /etc/racoon/racoon.conf

# Simple racoon.conf
#
#
# Please look in /usr/share/doc/racoon/examples for
# examples that come with the source.
#
# Please read racoon.conf(5) for details, and alsoread setkey(8).
#
#
# Also read the Linux IPSEC Howto up at
# http://www.ipsec-howto.org/t1.html
#

path pre_shared_key "/etc/racoon/psk.txt";
path certificate "/etc/racoon/certs";

remote 99.99.99.99 {
        exchange_mode aggressive,main;
        dpd_delay 20;
        dpd_maxfail 5;
        my_identifier user_fqdn "something@company.com";
        proposal {
                lifetime time 28800 second;
                encryption_algorithm 3des;
                hash_algorithm sha1;
                authentication_method pre_shared_key;
                dh_group modp1024;
        }
}

sainfo address 10.0.2.0/24 any address 10.0.1.0/24 any {
        lifetime time 28800 second;
        pfs_group modp1024;
        encryption_algorithm 3des;
        authentication_algorithm hmac_sha1;
        compression_algorithm deflate;
}

Watchguard side

I have simply taken screen shots for the configuration for the watchguard side:


You will need to ensure that the preshared random key is the same as you used in the /etc/racoon/psk.txt file you created above.






Tuesday, 19 March 2013

VMs, snapshots and domain computer accounts.

Have you ever had the problem where you have reverted a VM's snapshot to find that it's computer password is out of sync with the domain? I have, loads of times. This is often seen with the following message when you attempt to log in: "Windows cannot connect to the domain, either because the domain
controller is down or otherwise unavailable, or because your computer
account was not found."

The problem is that Windows machines on a Domain change their computer account passwords with a Domain controller every 30 Days. If a machine changes it's computer password with a domain controller and you then revert to a snapshot that was taken before the password was changed the computer account will no longer be able to authenticate on the network and domain users won't be able to logon.

How do you get around this? One way is in Microsoft's KB article 154501.

This can reduce the security on your domain, or at least the security between the DC and the workstation you make the following registry changes on but if you have a testing setup like I do this is not much of a problem and the convenience easily outweighs any security issues (in my opinion :).
You can set the DisablePasswordChange registry entry to 1 in :

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters

That has now stopped this machine from changing it's domain password every 30 days. Now we need to get the machine back onto the domain, to achieve this do one of the following:

A. Remove the computer from the domain and add it back in again. Easy enough you say. I have a lot of test machines floating about that don't always have the same local administrator password. So it is a good idea to make sure that you know a local administrators password before you remove the machine from the domain otherwise you will end up with a machine you cannot access via windows (all is not necessarily lost, trinity rescue kit can help here).

B. Remove and rejoin the machine to the domain using the netdom.exe command:
netdom remove <machine name> /Domain:<domain name> /userd:<domain name>\<domain administrator account name> /passwordd:<domain administrator password>
Wait for the response:
The command completed successfully.
Then run:
netdom join <machine name> /Domain:<domain name> /userd:<domain name>\<domain administrator account name> /passwordd:<domain administrator password>
Again we are looking for the response:
The command completed successfully.
It is probably best to reboot the machine at this point, it is a windows machine after all we were messing with the domain membership and I simply would trust a machine in this state to function as expected unless it is rebooted.

Wednesday, 14 November 2012

ESXi not accepting previously used volume

I recently had a problem on an ESXi 5.0 box I was trying to add a iscsi target to. I had set up a 6 TB  iSCSI LUN but everytime I tried adding it to ESXI I got the following error:

Call "HostDatastoreSystem.QueryVmfsDatastoreCreateOptions" for object "ha-datastoresystem" on ESXi "ESX HOSTNAME" failed. 

It turned out that the setup wizard on the device had happily created a volume and formatted it with ext4 before I realised what it was doing. Although I removed this volume the partition information wasn't removed from the array and this was upsetting ESXi.

How do you fix it? As there was no obvious way to wipe the partition info from within the Array's management interface I decided to do it from a Debian VM running on that host.

Firstly I removed the discovery information from the ESXI server so it isn't trying to interfere then on the Debian VM I installed open-iscsi with:

apt-get install open-iscsi

Start the open-iscsi daemon with:

/etc/init.d/open-iscsi start

Query the iSCSI target on the storage device:

iscsiadm -m discovery -t st -p 192.168.1.99

Which should return something like this:
192.168.1.99:3260,0 iqn.2010-12.com.manufacturer:nasdevice.name

If you have set up iSCSI authentication on your storage device you will need to run something like the following commands using the iqn in the above response.

iscsiadm   --mode node  --targetname "iqn.2010-12.com.manufacturer:nasdevice.name"  -p 192.168.0.99:3260 --op=update --name node.session.auth.authmethod --value=CHAP
iscsiadm   --mode node  --targetname "iqn.2010-12.com.manufacturer:nasdevice.name"  -p 192.168.0.99:3260 --op=update --name node.session.auth.username --value=username
iscsiadm   --mode node  --targetname "iqn.2010-12.com.manufacturer:nasdevice.name"  -p 192.168.0.99:3260 --op=update --name node.session.auth.password --value=password

Logon to the storage device:

iscsiadm -m node --targetname "iqn.2010-12.com.manufacturer:nasdevice.name" --portal "192.168.0.99:3260" --login

All being well this should now create a SCSI device as if you had attached a hard drive directly to the system. I looked at the bottom of the output from the dmesg command to find out which device (/dev/sdb).

I used the following command to write zeros to the first half a MB of the disk which will overwrite any partition table information (care should be taken that you have the correct device when using this command, it will eat your drive):

 dd if=/dev/zero of=/dev/sdb bs=512 count=1024

Now logout of the storage device with the following command:

iscsiadm -m node --targetname "iqn.2010-12.com.manufacturer:nasdevice.name" --portal "192.168.0.99:3260" --logout

ESXi should now happily accept the iSCSI LUN when you attempt to add it.

Monday, 12 November 2012

Local security policy on Windows 8

The Local Security Policy MMC in Windows 8 how now been moved further away from the user (perhaps with good cause) and is available by searching for "secpol.msc."

This is reminiscent of the move to hide the Component services mmc in Windows 7/Windows Server 2008 R2 which can me accessed by searching for "comexp.msc."

Tuesday, 6 November 2012

Installation of .net 3.5 on Windows Server 2012

I struggled installing .net 3.5 on Windows 2012 Server. After going through the add features and selecting .net 3.5 I got the following Warning:

The request to add or remove feature on the specified server failed.
Installation of one or more roles, role services, or features failed. - The source files could not be downloaded.
Use the /source option to specify the location of the files that are required to restore the feature. The file location should be either the root directory of a mounted image or a component store that has the Windows Side-by-Side directory as an immediate subfolder.

From a administrative command prompt run the following:

dism.exe /online /enable-feature /featurename:NetFX3 /Source:d:\sources\sxs /LimitAccess /all

This will install .net 2.0 and 3.5.

Friday, 13 July 2012

"Remote Downlevel Document" instead of jobname in Samba

To save the planet I implemented a print to PDF server for our Quality Assurance department at work. It consists an installation of Samba and a hand full of custom scripts.

I recently upgraded this server and added it to our domain (up until now there were no sensitive documents being processed but as we planned on adding these it became necessary to restrict access to some shares).

I have just spent far too much time trying to work out why windows was not passing the document name from the printing application correctly. It seems that the "jobname" parameter being passed from Samba to my script was always "Remote Downlevel Document" when previously it was something more helpful like the title of a document.

The offending line in the configuration was:

disable spoolss = yes

Once this was removed everything started working as before.

Thursday, 21 June 2012

Using screen on a linux server

Once in a while I need to leave a process running for a long period of time on a Linux server I will use the screen command. Screen allow you to run a process in a session then disconnect your SSH session but leave the process running, then log back in later and check in on the process from another SSH session.

Once screen has been installed (sudo apt-get install screen on a Debian system) you can simply type screen followed by the command line for the process you want to run. So if you want to leave a network trace running you would use something like:

# screen tcpdump -ieth0 host hostname.domain.com -w ./host.cap

This would start up tcpdump and run it in a screen session.

If you only have one screen session running on a machine then to get to it you can use:

# screen -r

However if you plan on running more than one session then you may want to give the sessions names, so for example you can run :

# screen -S capture-session tcpdump -ieth0 host hostname.domain.com -w ./host.cap


Then you can access that specific session with:


# screen -r capture-session


Once you are in a session it will perform like an ordinary SSH connection. If you wish to disconnect but leave the session running you will need to press <CTRL>+a, then d.

Tuesday, 19 June 2012

Installing Windows 8 x64 Release Preview on an ESXI VM

I simply thought that it would be a case of installing Windows 8 release preview and I would be able to have a tinker. Unfortunately not.

Firstly make sure you are on ESXi 5.0 update1. Create a virtual machine and set the OS to "Microsoft Windows 8 (64-bit)". At the end of creating the VM, us the check box"open setting dialog"

In the VM's settings:
  • Change the video card to Auto detect settings and check the "Enable 3D support"
  • go to the Options->Boot Options and change the firmware to EFI
Now proceed with the installation of Windows 8 Release Preview and all should work.

I didn't modify the settings on the VM the first time I tried to install Windows 8 and ended up having to reinstall after making the above changes.

Wednesday, 6 June 2012

MySQL Server Replication

Recently had cause to check out MySQL's replication capabilities. One of the programmers wanted a wiki for his department's to use. We already used Mediawiki on a Debian box for another department so I thought we could beef up the box with some RAM then add a new wiki to it, change the backup scripts and we would be done. However I got thinking that this might be an opportunity to provide a more resilient set up where the MySQL server running the wiki could replicate to an offsite location. This could then provide a read-only back up to the existing machine in case of failure. I will of course still need to take backups (just in case some Flump deletes the lot).

The following is how I set up replication between a production mediawiki server with the MySQL server installed locally.

It kind of goes without saying that you will need at least two machines each running MySQL, in my set up both machines were Debian 6 (Squeeze).

On the Production machine we will need to to configure MySQL so it knows it is going to be part of a replication group. Edit my.cnf and add or modify the following:

server-id = 1
log_bin = /var/log/mysql/mysql-bin.log

We also need to get MySQL to bind to the network address of the NIC

bind-address = 0.0.0.0

On the Offsite machine we need to tell it to be a slave:

server-id = 2
log_bin = /var/log/mysql/mysql-bin.log

We need to create an account on the master (production) machine for the slave to be able to log on to the master for replication.
$ mysql -u root -p
mysql>  CREATE USER 'replicationos'@'%' IDENTIFIED BY 'password'; 
Query OK, 0 rows affected (0.10 sec)

mysql> GRANT REPLICATION SLAVE ON *.* TO 'replicationos'@'%';
Query OK, 0 rows affected (0.00 sec)

We now need to get the Replication master binary log components. If this is done on a production machine then I suggest you do it quickly because you will stop certain transactions from occurring while the tables are locked.

open two sessions to mysql, in the first run:

mysql> FLUSH TABLES WITH READ LOCK;

In the second session run:

mysql> show master status;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000003 |    21381 |              |                  |
+------------------+----------+--------------+------------------+
1 row in set (0.02 sec)

Make a note of the"File" and "Postition":

Exit mysql client in the second session (still leaving the first logged in). Now dump the data out ready to be imported by the slave with:

$ mysqldump -p --all-databases --lock-all-tables >/root/dbdump.db

Copy the dbdump.db to the slave machine with scp then on the slave machine start mysql with the --skip-slave-start option:

mysqld --skip-slave-start

Import the data with:

mysql -p < /root/dbdump.db

Now we need to tell the slave machine about the master using the "File" and "Position" we noted in a earlier step with the following command:

mysql> CHANGE MASTER TO MASTER_HOST='wiki.company.com', MASTER_USER='replicationos', MASTER_PASSWORD='password', MASTER_LOG_FILE='mysql-bin.000003', MASTER_LOG_POS=21381;
Now start the slave process:
mysql> start slave;


Wait for a few moments then check the status of replication with the following:

mysql> SHOW SLAVE STATUS\G

*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: wiki.company.com
                  Master_User: replicationos
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000027
          Read_Master_Log_Pos: 37742
               Relay_Log_File: mysqld-relay-bin.000511
                Relay_Log_Pos: 7333
        Relay_Master_Log_File: mysql-bin.000027
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 37742
              Relay_Log_Space: 7489
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
1 row in set (0.00 sec)



The \G in the "SHOW SLAVE STATUS\G" is short hand for "ego" in the MySQL client and it means "Send command to mysql server, display result vertically." Check that the "Slave_IO_State" is "Waiting for master to send event" and that "Seconds_Behind_Master" is 0 and that means that your slave is synchronised with your server (unless you have only just lost network connection to the master).

Friday, 20 April 2012

Linux giving out wrong MAC address

I recently had a problem where a Linux box with multiple NICs running a iptables firewall script was not accepting connections when it should. I ran tcpdump to find that ARP lookups received by the box were actually giving out the a MAC address from a different NIC. This seemed bizarre at first, but knowing Linux's networking strength I realised that I was missing something and that I probably needed to tweak the kernel to behave the way I wanted.

A quick Google found that I probably wanted to turn on arp_filter in the kernel. So I ran the following:

echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter

Immediately tcpdump showed that the kernel was now behaving the way I wanted it to and handing out only the MAC address for the relevant NIC.

To make the change permanent (i.e. survive a reboot) you could add the above line to a init script or add the following line to /etc/sysclt.conf

net.ipv4.conf.all.arp_filter = 1

Tuesday, 17 April 2012

NTFS alternate data streams and zip files

If you download a zip file onto a NTFS partition Internet Explorer will store zone information in the file using something called alternate data streams. If the zone information is stored with the file it will be "blocked" within Windows. The file can easily be unblocked by clicking the "Unblock" button in the files properties (if your user account has the correct permissions).
File properties showing the unblock button
We recently had a problem where we were downloading a zip file to a customer's machine, unzipping the archive and when we tried to run the executable that was in the archive it was failing. It seems that if you do not "unblock" the zip file then all of the files extracted from the archive will all be in the same blocked state. This was causing problems for our support team and it seems the answer to this problem is to simply unblock the zip before unpacking the archive (I suppose you could unblock each individual file after the extraction but we had a lot of files).

To see this behaviour download a file with IE and look at the properties. The alternate data stream is called Zone.Identifier, you can see the contents of the alternate data stream with the following command:

more < fullfilename:Zone.Identifier

Seeing the zone information saved with the alternate data stream on a downloaded file


Friday, 24 February 2012

Copy and paste in the vSphere client

I assumed that the ability to copy and paste text between your desktop and virtual machines had been removed entirely from the vSphere client. It turns out that it has just been disabled by default for security reasons and that it can be enabled per VM or per ESX host.

I found a need for this when setting up machines with little or no contact with the outside world, but that you want to make complicated configuration changes that you have previously documented.

Information on how to enable copy and paste in the vSphere client is available in VMware's KB article 1026437.

Friday, 16 December 2011

Converting a Debian VM from Xenserver to vSphere

You can transfer the data for the machine from one environment to another using clonezilla. Once it is transferred I found that the machine does not boot. On further investigation I realised that there are many differences between a Debian install on Xenserver and on vSphere.

If you get the message "Operating system not found" you will need to install grub to the virtual hard disk. To do this boot a liveCD (such as clonezilla).

As root run grub, at the grub> prompt type:

grub> find /boot/grub/stage1

This will return the location of grubs stage one, make a note of this location for the next command. Type

grub> root (hd0,0) < this is the location returned by the last command

Now install grub to the master boot record (MBR) with:

grub> setup (hd0)

then quit grub with:

grub> quit

Reboot to get the grub menu but if you let it try and continue using the default settings it will very likely hang because it will try to use a block device name that begins with xvd, these are xenserver specific and will not exist on the VMware VM. To get it to continue booting you need to hit 'e' on the grub selection screen, then edit the boot line to replace the device that looks like /dev/xvda1 and made it look like /dev/sda1, also remove the "console=hvc0" otherwise you will not be able to see interact with your OS via the console (hvc0 is another xenserver specific device name).

Once we have booted the system we need to go about preparing the system to boot correctly when left to it's own devices (geddit?), firstly ensure the keyboard keymap is set up correctly with:

# dpkg-reconfigure console-data

Edit /etc/fstab to change all references to xvd devices to sda in /etc/fstab

On many of the machines I transferred using this method I found that they had no swap partitions after the transfer. So I simply set the disk size slightly bigger than the source disk when initially creating the VM in vSphere, then added a partition with fdisk. I usually set the swap partition as /dev/sda2.
You then need to make that partition a swap partition with (I have found often that a reboot is required before the command works):

# mkswap /dev/sda2

Then you can mount it with:

# swapon /dev/sda2

And see if the swap space is available with:

# free

Don't forget to update /etc/fstab so it is pointing to the correct device for the swap partition so it is automounted at boot.

The CDRom is set to /dev/hdc unless you have changed the VM's hardware settings, update this in /etc/fstab as well.

We need to change grubs configuration so that it automatically boots a with appropriate parameters, this information is stored in /boot/grub/menu.list. You can achieve this by either running the following commands:

sed -i 's/xvd/sd/g' /boot/grub/menu.lst
sed -i 's/console=hvc0//g' /boot/grub/menu.lst

Or edit  /boot/grub/menu.lst, find the line that begins # kopt, remove the console=hvc0 from the end and change the "xvd" device to the correct boot device. move down to the botton where the menu choices are configured and change the device and remove the console=hvc0 from each of the one you will likely use (the top one only in my case).

Edit /etc/inittab and comment out the line that begins "co:"

Remove the xenserver specific packages from your apt sources list with:

rm /etc/apt/sources.list.d/citrix.list

Update your packages list:

apt-get update

Remove all the kernels that are installed with:

apt-get remove linux-image*

You will get a warning asking about the removal of the running linux kernel, say "No" to this as we will install another one that we can get headers for in the next step.

Install a more appropriate kernel with:

apt-get install linux-image-2.6-686

Reboot to run the newly installed kernel.

Install VMware tools, details of how in my post here, then reboot to test.

To get the most out of the VM you should switch hardware to paravirutal drivers for SCSI controllers and NICs.


Thursday, 15 December 2011

Convert a Windows machine to a VMware VM with Clonezilla

I have had need to do manual conversions of physical and virtual machines for several reasons previously. Some of these reasons have included:
  • There is not enough free space on the machine's disk(s), (also know as "I haven't got time to wait for the machine's administrator to tidy up")
  • VMware Convert no longer supports the conversion of that operating system.
  • We can't afford PlateSpin
  • I wonder if I could do a conversion the hard way.
This takes me back to my first ESX Project. The company I worked for had about 8 test machines sitting along one office wall. The situation was already out of hand and I was being asked for more test machines. I decided we needed a to think big and decided on and got approved ESX on a couple of big (at least for us) servers to virtualise the test environment. I managed to simply move all the test machines by installing a fresh OS and installing and configuring all the software my self, with the help of the relevant departments. I then came to our Windows NT 4.0 software build machine.... No-one, not even the developer that put it together knew what was on there. We were building releases every couple of weeks at the time. It goes without saying this is not a good place to be, and baring a complete hardware failure on the physical machine there was no way anyone was going to be rebuilding the machine. I scratched my head and had previously used dd and tar to move installs from one machine to another on Linux, so I had a crack at it, and to my surprise a few hours later I had run a P2V by hand with a Linux live CD. I had even managed convince Windows NT to let me switch the drives from IDE to SCSI.

Having used Clonezilla to store a few images from machines that were shipped without restore media in the past, I wondered how easy it would be to back up an image of a machine and restore it into a new virtual environment. As it turns out with Clonezilla I don't even need to create an image, the tools are there to transfer the data from one machine's disk and write it directly to the disk of another.

The conversion here is from a Xenserver VM to a VMware VM but the technique used here will probably work else where. 
  • Put a Clonezilla live CD in both your (virtual) machines and boot them.
  • The default boot option for Clonezilla worked fine for me on both machines, your mileage may vary.
  • Select your language options as necessary
On the source machine:
  • Select "Start_Clonezilla"
  • Select "device-device"
  • Select "Beginner"
  • Select "disk_to_remote_disk"
  • Select "dhcp" or assign an IP address with "static"
  • Now it should show you a list of disks in the system, select the one you wish to transfer.
  • Select "Skip checking/repairing source file system" 
  • Clonezilla will now show you the command it is about to run, press "Enter."
  • You will be asked to confirm at a few important stages that you wish to proceed.
  • When you see "Waiting for the target machine to connect..." you have finished with the source machine for now, except take a note of the commands it is telling you run on the destination machine (See below image.)
On the destination machine:
  • Select "Enter_shell"
  • Select (2)
  • Type: sudo su - (to change to the root user)
  • Type: ocs-live-netcfg (and setup the networking)
  • Type: ocs-onthefly -s 192.168.1.1 -t sda  (replacing 192.168.1.1 and sda with the relevant IP address and device name)
  • You will again be asked to confirm at poignant stages.
You should see a progress bar which includes an estimate for the finish time.


When this finishes you should now be able to boot the destination VM and install the VMWare tools to make everything pretty and efficient. Don't forget to install the paravirtual SCSI and network card drivers if your OS is supported.


If you have success (or failure) with this migration method please leave a comment below, it certainly helped me out of a hole.