ESXi not accepting previously used volume

I recently had a problem on an ESXi 5.0 box I was trying to add a iscsi target to. I had set up a 6 TB  iSCSI LUN but everytime I tried adding it to ESXI I got the following error:

Call "HostDatastoreSystem.QueryVmfsDatastoreCreateOptions" for object "ha-datastoresystem" on ESXi "ESX HOSTNAME" failed. 

It turned out that the setup wizard on the device had happily created a volume and formatted it with ext4 before I realised what it was doing. Although I removed this volume the partition information wasn't removed from the array and this was upsetting ESXi.

How do you fix it? As there was no obvious way to wipe the partition info from within the Array's management interface I decided to do it from a Debian VM running on that host.

Firstly I removed the discovery information from the ESXI server so it isn't trying to interfere then on the Debian VM I installed open-iscsi with:

apt-get install open-iscsi

Start the open-iscsi daemon with:

/etc/init.d/open-iscsi start

Query the iSCSI target on the storage device:

iscsiadm -m discovery -t st -p 192.168.1.99

Which should return something like this:
192.168.1.99:3260,0 iqn.2010-12.com.manufacturer:nasdevice.name

If you have set up iSCSI authentication on your storage device you will need to run something like the following commands using the iqn in the above response.

iscsiadm   --mode node  --targetname "iqn.2010-12.com.manufacturer:nasdevice.name"  -p 192.168.0.99:3260 --op=update --name node.session.auth.authmethod --value=CHAP
iscsiadm   --mode node  --targetname "iqn.2010-12.com.manufacturer:nasdevice.name"  -p 192.168.0.99:3260 --op=update --name node.session.auth.username --value=username
iscsiadm   --mode node  --targetname "iqn.2010-12.com.manufacturer:nasdevice.name"  -p 192.168.0.99:3260 --op=update --name node.session.auth.password --value=password

Logon to the storage device:

iscsiadm -m node --targetname "iqn.2010-12.com.manufacturer:nasdevice.name" --portal "192.168.0.99:3260" --login

All being well this should now create a SCSI device as if you had attached a hard drive directly to the system. I looked at the bottom of the output from the dmesg command to find out which device (/dev/sdb).

I used the following command to write zeros to the first half a MB of the disk which will overwrite any partition table information (care should be taken that you have the correct device when using this command, it will eat your drive):

 dd if=/dev/zero of=/dev/sdb bs=512 count=1024

Now logout of the storage device with the following command:

iscsiadm -m node --targetname "iqn.2010-12.com.manufacturer:nasdevice.name" --portal "192.168.0.99:3260" --logout

ESXi should now happily accept the iSCSI LUN when you attempt to add it.

Local security policy on Windows 8

The Local Security Policy MMC in Windows 8 how now been moved further away from the user (perhaps with good cause) and is available by searching for "secpol.msc."

This is reminiscent of the move to hide the Component services mmc in Windows 7/Windows Server 2008 R2 which can me accessed by searching for "comexp.msc."

Installation of .net 3.5 on Windows Server 2012

I struggled installing .net 3.5 on Windows 2012 Server. After going through the add features and selecting .net 3.5 I got the following Warning:

The request to add or remove feature on the specified server failed.
Installation of one or more roles, role services, or features failed. - The source files could not be downloaded.
Use the /source option to specify the location of the files that are required to restore the feature. The file location should be either the root directory of a mounted image or a component store that has the Windows Side-by-Side directory as an immediate subfolder.

From a administrative command prompt run the following:

dism.exe /online /enable-feature /featurename:NetFX3 /Source:d:\sources\sxs /LimitAccess /all

This will install .net 2.0 and 3.5.

"Remote Downlevel Document" instead of jobname in Samba

To save the planet I implemented a print to PDF server for our Quality Assurance department at work. It consists an installation of Samba and a hand full of custom scripts.

I recently upgraded this server and added it to our domain (up until now there were no sensitive documents being processed but as we planned on adding these it became necessary to restrict access to some shares).

I have just spent far too much time trying to work out why windows was not passing the document name from the printing application correctly. It seems that the "jobname" parameter being passed from Samba to my script was always "Remote Downlevel Document" when previously it was something more helpful like the title of a document.

The offending line in the configuration was:

disable spoolss = yes

Once this was removed everything started working as before.

Using screen on a linux server

Once in a while I need to leave a process running for a long period of time on a Linux server I will use the screen command. Screen allow you to run a process in a session then disconnect your SSH session but leave the process running, then log back in later and check in on the process from another SSH session.

Once screen has been installed (sudo apt-get install screen on a Debian system) you can simply type screen followed by the command line for the process you want to run. So if you want to leave a network trace running you would use something like:

# screen tcpdump -ieth0 host hostname.domain.com -w ./host.cap

This would start up tcpdump and run it in a screen session.

If you only have one screen session running on a machine then to get to it you can use:

# screen -r

However if you plan on running more than one session then you may want to give the sessions names, so for example you can run :

# screen -S capture-session tcpdump -ieth0 host hostname.domain.com -w ./host.cap


Then you can access that specific session with:


# screen -r capture-session


Once you are in a session it will perform like an ordinary SSH connection. If you wish to disconnect but leave the session running you will need to press <CTRL>+a, then d.

Installing Windows 8 x64 Release Preview on an ESXI VM

I simply thought that it would be a case of installing Windows 8 release preview and I would be able to have a tinker. Unfortunately not.

Firstly make sure you are on ESXi 5.0 update1. Create a virtual machine and set the OS to "Microsoft Windows 8 (64-bit)". At the end of creating the VM, us the check box"open setting dialog"

In the VM's settings:

• Change the video card to Auto detect settings and check the "Enable 3D support"
• go to the Options->Boot Options and change the firmware to EFI

Now proceed with the installation of Windows 8 Release Preview and all should work.

I didn't modify the settings on the VM the first time I tried to install Windows 8 and ended up having to reinstall after making the above changes.

MySQL Server Replication

Recently had cause to check out MySQL's replication capabilities. One of the programmers wanted a wiki for his department's to use. We already used Mediawiki on a Debian box for another department so I thought we could beef up the box with some RAM then add a new wiki to it, change the backup scripts and we would be done. However I got thinking that this might be an opportunity to provide a more resilient set up where the MySQL server running the wiki could replicate to an offsite location. This could then provide a read-only back up to the existing machine in case of failure. I will of course still need to take backups (just in case some Flump deletes the lot).

The following is how I set up replication between a production mediawiki server with the MySQL server installed locally.

It kind of goes without saying that you will need at least two machines each running MySQL, in my set up both machines were Debian 6 (Squeeze).

On the Production machine we will need to to configure MySQL so it knows it is going to be part of a replication group. Edit my.cnf and add or modify the following:

server-id = 1
log_bin = /var/log/mysql/mysql-bin.log

We also need to get MySQL to bind to the network address of the NIC

bind-address = 0.0.0.0

On the Offsite machine we need to tell it to be a slave:

server-id = 2
log_bin = /var/log/mysql/mysql-bin.log

We need to create an account on the master (production) machine for the slave to be able to log on to the master for replication.

$ mysql -u root -p

mysql>  CREATE USER 'replicationos'@'%' IDENTIFIED BY 'password'; 

Query OK, 0 rows affected (0.10 sec)

mysql> GRANT REPLICATION SLAVE ON *.* TO 'replicationos'@'%';

Query OK, 0 rows affected (0.00 sec)

We now need to get the Replication master binary log components. If this is done on a production machine then I suggest you do it quickly because you will stop certain transactions from occurring while the tables are locked.

open two sessions to mysql, in the first run:

mysql> FLUSH TABLES WITH READ LOCK;

In the second session run:

mysql> show master status;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000003 |    21381 |              |                  |
+------------------+----------+--------------+------------------+
1 row in set (0.02 sec)

Make a note of the"File" and "Postition":

Exit mysql client in the second session (still leaving the first logged in). Now dump the data out ready to be imported by the slave with:

$ mysqldump -p --all-databases --lock-all-tables >/root/dbdump.db

Copy the dbdump.db to the slave machine with scp then on the slave machine start mysql with the --skip-slave-start option:

mysqld --skip-slave-start

Import the data with:

mysql -p < /root/dbdump.db

Now we need to tell the slave machine about the master using the "File" and "Position" we noted in a earlier step with the following command:

mysql> CHANGE MASTER TO MASTER_HOST='wiki.company.com', MASTER_USER='replicationos', MASTER_PASSWORD='password', MASTER_LOG_FILE='mysql-bin.000003', MASTER_LOG_POS=21381;
Now start the slave process:
mysql> start slave;


Wait for a few moments then check the status of replication with the following:

mysql> SHOW SLAVE STATUS\G

*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: wiki.company.com
                  Master_User: replicationos
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000027
          Read_Master_Log_Pos: 37742
               Relay_Log_File: mysqld-relay-bin.000511
                Relay_Log_Pos: 7333
        Relay_Master_Log_File: mysql-bin.000027
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 37742
              Relay_Log_Space: 7489
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
1 row in set (0.00 sec)

The \G in the "SHOW SLAVE STATUS\G" is short hand for "ego" in the MySQL client and it means "Send command to mysql server, display result vertically." Check that the "Slave_IO_State" is "Waiting for master to send event" and that "Seconds_Behind_Master" is 0 and that means that your slave is synchronised with your server (unless you have only just lost network connection to the master).

Linux giving out wrong MAC address

I recently had a problem where a Linux box with multiple NICs running a iptables firewall script was not accepting connections when it should. I ran tcpdump to find that ARP lookups received by the box were actually giving out the a MAC address from a different NIC. This seemed bizarre at first, but knowing Linux's networking strength I realised that I was missing something and that I probably needed to tweak the kernel to behave the way I wanted.

A quick Google found that I probably wanted to turn on arp_filter in the kernel. So I ran the following:

echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter

Immediately tcpdump showed that the kernel was now behaving the way I wanted it to and handing out only the MAC address for the relevant NIC.

To make the change permanent (i.e. survive a reboot) you could add the above line to a init script or add the following line to /etc/sysclt.conf

net.ipv4.conf.all.arp_filter = 1

NTFS alternate data streams and zip files

If you download a zip file onto a NTFS partition Internet Explorer will store zone information in the file using something called alternate data streams. If the zone information is stored with the file it will be "blocked" within Windows. The file can easily be unblocked by clicking the "Unblock" button in the files properties (if your user account has the correct permissions).

File properties showing the unblock button

We recently had a problem where we were downloading a zip file to a customer's machine, unzipping the archive and when we tried to run the executable that was in the archive it was failing. It seems that if you do not "unblock" the zip file then all of the files extracted from the archive will all be in the same blocked state. This was causing problems for our support team and it seems the answer to this problem is to simply unblock the zip before unpacking the archive (I suppose you could unblock each individual file after the extraction but we had a lot of files).

To see this behaviour download a file with IE and look at the properties. The alternate data stream is called Zone.Identifier, you can see the contents of the alternate data stream with the following command:

more < fullfilename:Zone.Identifier

Seeing the zone information saved with the alternate data stream on a downloaded file

Copy and paste in the vSphere client

I assumed that the ability to copy and paste text between your desktop and virtual machines had been removed entirely from the vSphere client. It turns out that it has just been disabled by default for security reasons and that it can be enabled per VM or per ESX host.

I found a need for this when setting up machines with little or no contact with the outside world, but that you want to make complicated configuration changes that you have previously documented.

Information on how to enable copy and paste in the vSphere client is available in VMware's KB article 1026437.