I - Cài đặt và cấu hình Proxmox:
- Download file ISO tại trang chủ: http://www.proxmox.com.
- Sau khi cài đặt thành công proxmox, chúng ta có thể truy cập vào giao diện web tại địa chỉ https://<ip-server>:8006
- Mặc định sau khi cài đặt xong Proxmox địa chỉ IP public sẽ được gán trên card vmbr0, card này được bridge với card eth0 của server. Chúng ta cần thay đổi một số thông tin tại đây.
/etc/network/interfaces
Địa chỉ IP public của server sẽ được gán trên card eth0. Đối với card vmbr0 chúng ta sẽ chuyển bridge_ports sang none (thay vì eth0) trước đó. Cấu hình mẫu như bên dưới
# network interface settings
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address 192.168.100.115
netmask 255.255.255.0
gateway 192.168.100.2
dns-nameservers 8.8.8.8
auto vmbr0
iface vmbr0 inet static
address 10.10.10.1
netmask 255.255.255.0
bridge_ports none
bridge_stp off
bridge_fd 0
Giả sử ở cấu hình trên, địa chỉ public của server sẽ là 192.168.100.115 được gán lên card eth0, còn địa chỉ 10.10.10.1 được gán trên card vmbr0 sẽ giao tiếp với các VM bên trong.
Trên giao diện web của Proxmox, thẻ network ta sẽ có thông tin như sau

Chúng ta cũng sẽ setup một VM để test. VM này sẽ có địa chỉ IP là 10.10.10.10.

II - Cài đặt và cấu hình shorewall
- Cài đặt shorewall bằng lệnh: apt-get install shorewall
- Kích hoạt Ip forwarding trên shorewall bằng cách mở file /etc/shorewall/shorewall.conf
Đổi
IP_FORWARDING=Off
Thành
IP_FORWARDING=On
- Cấu hình Zone trong file /etc/shorewall/zones. Chúng ta sẽ có 3 Zone fw, net, và dmz. Trong đó net là zone giao tiếp từ môi trường internet, dmz là zone nội bộ bên trong.
#ZONE TYPE OPTIONS IN OUT
# OPTIONS OPTIONS
fw firewall
net ipv4
dmz ipv4
- Cho biết các zone được kết nối thông qua interface nào. Tạo file /etc/shorewall/interfaces
#ZONE INTERFACE BROADCAST OPTIONS
net eth0 detect blacklist,nosmurfs
dmz venet0 detect routeback
dmz vmbr0 detect routeback,bridge
- Tiếp tục, chúng ta cấu hình các policy mặc định giữa các zone. Tạo file /etc/shorewall/policy
#SOURCE DEST POLICY LOG LIMIT: CONNLIMIT:
# LEVEL BURST MASK
# From Firewall Policy
fw fw ACCEPT
fw net ACCEPT
fw dmz ACCEPT
# From DMZ Policy
dmz dmz ACCEPT
dmz net ACCEPT
dmz fw DROP info
# From Net Policy
net fw DROP info
net dmz DROP info
# THE FOLLOWING POLICY MUST BE LAST
#
all all REJECT info
- Thiết lập rules cho shorewall tại /etc/shorewall/rules
#ACTION SOURCE DEST PROTO DEST SOURCE ORIGINAL RATE
# Permit access to SSH
SSH/ACCEPT net fw - - - - 6/min:5
# Permit access to Proxmox Manager and Console
ACCEPT net fw tcp 8006,5900:5999
# VM access rules
DNAT net dmz:10.10.10.10:22 tcp 1022
# PING Rules
Ping/ACCEPT all all
# LAST LINE -- DO NOT REMOVE
Trong ví dụ về cấu hình rule ở trên, chúng ta cho phép SSH và ping tới Proxmox server. Ngoài ra, chúng ta còn thực hiện việc NAT port 22 trên VM. Với cấu hình này, nếu chúng ta muốn SSH tới VM từ bên ngoài internet, chúng ta có thể SSH thông qua public IP với port là 1022 (Cấu hình cho các dịch vụ khác tương tự).
- Thiết lập cho phép kết nối từ VM ra ngoài internet thông qua tệp tin /etc/shorewall/masq
#INTERFACE SOURCE ADDRESS PROTO PORT(S) IPSEC MARK
eth0 10.10.10.0/24
# LAST LINE -- DO NOT REMOVE
- Kiểm tra cấu hình của shorewall đã đúng hay chưa
shorewall check
- Nếu mọi cấu hình đều OK, chúng ta kích hoạt cho phép Shorewall được khởi động
- Khởi động Shorewall
Chú ý: Nếu bạn SSH tới Proxmox server từ xa. Việc cấu hình sai shorewall có thể làm bạn bị mất kết nối khi khởi động dịch vụ shorewall. Để tránh tình trạng này, chúng ta có thể sử dụng lệnh sau để kiểm tra trước. Với lệnh này, shorewall sẽ chỉ apply cấu hình trong vòng 60 giây, sau thời gian đó, nó sẽ tự động khôi phục lại trạng thái trước đó.
shorewall try /etc/shorewall 60
Mọi cấu hình đã xong, bây giờ chúng ta có thể truy xất SSH tới VM bên trong thông qua ip public của Proxmox Server.
Install puppet master
master$ sudo apt-get -y install puppetmaster
Regenerate Master Certificate
master$ sudo service puppetmaster stop
master$ sudo rm -r /var/lib/puppet/ssl
Add master’s name to config
master$ sudoedit /etc/puppet/puppet.conf
add these names under [master] heading
dns_alt_names = puppet, master.local, puppet.khanh.com.vn
Certificate is automatically generated when you start PuppetMaster
master$ sudo service puppetmaster start
Connect from Slave
slave$ sudo apt-get -y install puppet
slave$ sudoedit /etc/puppet/puppet.conf
Add master DNS name under [agent] heading. Puppet will connect to server.
[agent]
server = master.local
Allow puppet slave to start
slave$ sudoedit /etc/default/puppet
Change to yes:
START=yes
Sign Slave Certificate on Master
master$ sudo puppet cert --list
master$ sudo puppet cert --sign slave.example.com
Create Site Manifest and a Module
....
Test Slave get configuration from Master
slave$ sudo puppet agent --test
------------------------
Force agent to regenerate certificate request by generate command
[puppet agent]$ puppet certificate generate web1 --ca-location remote
warning: peer certificate won't be verified in this SSL session
err: Error 400 on SERVER: web1 already has a revoked certificate; ignoring certificate request
err: Try 'puppet help certificate generate' for usage
#It because the revoked certificate still exist in the server, it need to be deleted
[puppet master]$ puppet cert list –all
- web1 (BA:18:D1:86:D6:5E:9E:99:55:39:3D:67:79:BF:BD:D0) (certificate revoked)
[puppet master]$ puppet cert clean web1
#re-run the command, the warning is expected because the request hasn’t been signed by master yet
[puppet agent]$puppet certificate generate web1 --ca-location remote
warning: peer certificate won't be verified in this SSL session
true
#The pending request appears in master
[puppet master]$ puppet cert list
web1 (3B:ED:D9:8D:2F:C2:A1:D3:89:B4:D0:FD:41:7E:5E:0C)
#Sign the certificate
[puppet master]# puppet cert sign web1
PROBLEM DESCRIPTION:
While installing Exchange 2010 on Windows 2008 R2 some times admins can see two error messages
Errors:
- Unable to read data from the Metabase. Ensure that Microsoft Internet Information Services is installed.
- Unable to access the ‘Default Web Site’ on this computer.
CAUSE: Exchange 2010 Prerequisites are not met successfully
SOLUTION: One can follow Microsoft Article to make sure all prerequisites are met:
http://technet.microsoft.com/en-us/library/bb691354(EXCHG.140).aspx
If all common roles are being installed on one server as I had to do some time back for test environment following is what is needed to be executed from Windows Powershell (should be started with "Run As Administrator")
- Import-Module ServerManager
- Add-WindowsFeature NET-Framework,RSAT-ADDS,Web-Server,Web-Basic-Auth,Web-Windows-Auth,Web-Metabase,Web-Net-Ext,Web-Lgcy-Mgmt-Console,WAS-Process-Model,RSAT-Web-Server,Web-ISAPI-Ext,Web-Digest-Auth,Web-Dyn-Compression,NET-HTTP-Activation,RPC-Over-HTTP-Proxy -Restart
- Set-Service NetTcpPortSharing -StartupType Automatic
Download sysbench from: http://sourceforge.net/projects/sysbench/
Sysbench need mysql-devel installed
# yum install mysql-devel
Following commands to install sysbench
# aclocal
# libtoolize --force --copy
sysbench_installation_root_folder > ./autogen.sh
sysbench_installation_root_folder > ./configure
sysbench_installation_root_folder > make
sysbench_installation_root_folder > make install
Example: Benchmark MySQL
1. Create test table in database test with 1.000.000 records
# sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=test --mysql-user=root --mysql-password=mypassword prepare
2. Benchmark MySQL follow command
# sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=test --mysql-user=root --mysql-password=mypassword --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run
3. Clean up DB afterwards
# sysbench --test=oltp --mysql-db=test --mysql-user=root --mysql-password=mypassword cleanup
B1: Cài đặt Tomcat
Cài đặt Tomcat phân chia thành các thư mục khác nhau, ví dụ /var/tomcat/serverA và /var/tomcat/serverB. Trong đó các thư mục serverA, serverB cần có những thư mục nội dung và cấu hình sau: bin, lib, conf, logs, temp, webapps, work
Xem thêm: Cách cài đặt Tomcat7
B2: Thay đổi port
Mỗi Instance sẽ lắng nghe trên các port khác nhau, cần phải thay đổi để không bị xung đột. Có 3 port cần phải đổi là
- shutdown port: Port này được dùng bở tiến trình java khi tomcat chạy. Khi script shutdown.sh được thực hiện, tomcat sẽ gửi tín hiện kết thúc thông qua port này. Ví dụ 8005
- connector port: Port dùng để kết nối tới tomcat từ xa (trình duyệt). Ví dụ 8080
- ajp port: Port dùng khi apache httpd server liên lạc với tomcat, sử dụng để tạo loadbalancer. Ví dụ: 8009
Bạn cần sử file conf/server.xml thay đổi port
B3: Tạo init script cho mỗi Instance
#!/bin/bash
# description: Tomcat Start Stop Restart
# processname: tomcat
# chkconfig: 234 20 80
JAVA_HOME=/usr/java/jdk1.7.0_17/
export JAVA_HOME
PATH=$JAVA_HOME/bin:$PATH
export PATH
CATALINA_HOME=/usr/share/tomcat7
case $1 in
start)
sh $CATALINA_HOME/bin/startup.sh
;;
stop)
sh $CATALINA_HOME/bin/shutdown.sh
;;
restart)
sh $CATALINA_HOME/bin/shutdown.sh
sh $CATALINA_HOME/bin/startup.sh
;;
esac
exit 0
------------------------
#!/bin/bash
# description: Tomcat Start Stop Restart
# processname: tomcat
# chkconfig: 234 20 80
JAVA_HOME=/usr/java/jdk1.7.0_17/
export JAVA_HOME
PATH=$JAVA_HOME/bin:$PATH
export PATH
CATALINA_BASE=/usr/share/tomcat7-new
case $1 in
start)
sh $CATALINA_BASE/bin/startup.sh
;;
stop)
sh $CATALINA_BASE/bin/shutdown.sh
;;
restart)
sh $CATALINA_BASE/bin/shutdown.sh
sh $CATALINA_BASE/bin/startup.sh
;;
esac
exit 0
This post will cover installing and basic configuration of apache-tomcat-7.0.30
If you do not already have the Java Development Kit (JDK) installed on your machine, you will need to download and install the required JDK for your platform.
If you do have the JDK installed, you can skip to: Step 2: Download and Install the apache-tomcat-7.0.30
Step 1: Install the JDK
[root@station1 Downloads]# rpm -ivh jdk-7u7-linux-x64.rpm
-->
[root@station1 Downloads]# JAVA_HOME=/usr/java/jdk1.7.0_17
[root@station1 Downloads]# export JAVA_HOME
[root@station1 Downloads]# PATH=$JAVA_HOME/bin:$PATH
[root@station1 Downloads]# export PATH
[root@station1 Downloads]# vim /root/.bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
JAVA_HOME=/usr/java/jdk1.7.0_17/
export JAVA_HOME
PATH=$JAVA_HOME/bin:$PATH
export PATH
#PATH=$PATH:$HOME/bin
#export PATH
Restart or logout and login
[root@station1 Downloads]# echo $JAVA_HOME
/usr/java/jdk1.7.0_17/
Step 2: Download and Unpack apache-tomcat-7.0.30 (or latest)
[root@station1 Downloads]# tar -xzvf apache-tomcat-7.0.30.tar.gz
[root@station1 Downloads]# mv apache-tomcat-7.0.30 /usr/share/
[root@station1 Downloads]# vim /etc/init.d/tomcat
#!/bin/bash
# description: Tomcat Start Stop Restart
# processname: tomcat
# chkconfig: 234 20 80
JAVA_HOME=/usr/java/jdk1.7.0_17
export JAVA_HOME
PATH=$JAVA_HOME/bin:$PATH
export PATH
CATALINA_HOME=/usr/share/apache-tomcat-7.0.30
case $1 in
start)
sh $CATALINA_HOME/bin/startup.sh
;;
stop)
sh $CATALINA_HOME/bin/shutdown.sh
;;
restart)
sh $CATALINA_HOME/bin/shutdown.sh
sh $CATALINA_HOME/bin/startup.sh
;;
esac
exit 0
[root@station1 ~]# chmod 755 /etc/init.d/tomcat
[root@station1 ~]# chkconfig --add tomcat
[root@station1 ~]# chkconfig --level 234 tomcat on
[root@station1 ~]# chkconfig --list tomcat
tomcat 0:off 1:off 2:on 3:on 4:on 5:off 6:off
[root@station1 ~]# service tomcat stop
Using CATALINA_BASE: /usr/share/apache-tomcat-7.0.30
Using CATALINA_HOME: /usr/share/apache-tomcat-7.0.30
Using CATALINA_TMPDIR: /usr/share/apache-tomcat-7.0.30/temp
Using JRE_HOME: /usr/java/jdk1.7.0_17
Using CLASSPATH: /usr/share/apache-tomcat-7.0.30/bin/bootstrap.jar:/usr/share/apache-tomcat-7.0.30/bin/tomcat-juli.jar
[root@station1 ~]# service tomcat start
Using CATALINA_BASE: /usr/share/apache-tomcat-7.0.30
Using CATALINA_HOME: /usr/share/apache-tomcat-7.0.30
Using CATALINA_TMPDIR: /usr/share/apache-tomcat-7.0.30/temp
Using JRE_HOME: /usr/java/jdk1.7.0_17
Using CLASSPATH: /usr/share/apache-tomcat-7.0.30/bin/bootstrap.jar:/usr/share/apache-tomcat-7.0.30/bin/tomcat-juli.jar
[root@station1 ~]# service tomcat restart
Using CATALINA_BASE: /usr/share/apache-tomcat-7.0.30
Using CATALINA_HOME: /usr/share/apache-tomcat-7.0.30
Using CATALINA_TMPDIR: /usr/share/apache-tomcat-7.0.30/temp
Using JRE_HOME: /usr/java/jdk1.7.0_17
Using CLASSPATH: /usr/share/apache-tomcat-7.0.30/bin/bootstrap.jar:/usr/share/apache-tomcat-7.0.30/bin/tomcat-juli.jar
Using CATALINA_BASE: /usr/share/apache-tomcat-7.0.30
Using CATALINA_HOME: /usr/share/apache-tomcat-7.0.30
Using CATALINA_TMPDIR: /usr/share/apache-tomcat-7.0.30/temp
Using JRE_HOME: /usr/java/jdk1.7.0_07
Using CLASSPATH: /usr/share/apache-tomcat-7.0.30/bin/bootstrap.jar:/usr/share/apache-tomcat-7.0.30/bin/tomcat-juli.jar
For logs
[root@station1 ~]# less /usr/share/apache-tomcat-7.0.30/logs/catalina.out
We can now access the Tomcat Manager page at http://127.0.0.1:8080
------------
his is for dual boot os users, there are many who prefers dual boot to keep another OS optional. For any reason if you wish to change boot sequence unlike grub, grub2 requires couple of commands to run.
Here will take example to keep Windows as default boot.
First of all find menuentry for Windows
cat/boot/grub2/grub.cfg |grep Windows
|
result:-
menuentry "Windows 7 (loader) (on /dev/sda1)"--class windows --class os {
|
Set Windows menuentry as default (only entry mentioned in either " or 'from above command)
grub2-set-default "Windows 7 (loader) (on /dev/sda1)"
|
Verify default entry
Generate updated grub.cfg
grub2-mkconfig -o/boot/grub2/grub.cfg
|
Below is to set Fedora as default boot
cat /boot/grub2/grub.cfg |grep Fedora
grub2-set-default "Fedora Linux, with Linux 3.1.0-5.fc16.i686"
grub2-editenv list
grub2-mkconfig -o /boot/grub2/grub.cfg
|
Note: These steps only apply to EXT3 file systems.
Caution: VMware recommends having a complete backup of the virtual machine prior making these changes.
- Power off the virtual machine.
- Edit the virtual machine settings and extend the virtual disk size. For more information, see Increasing the size of a virtual disk (1004047).
- Power on the virtual machine.
- Identify the device name, which is by default /dev/sda, and confirm the new size by running the command:
# fdisk -l
-
Create a new primary partition:
- Run the command:
# fdisk /dev/sda (depending the results of the step 4)
- Press p to print the partition table to identify the number of partitions. By default there are 2: sda1 and sda2.
- Press n to create a new primary partition.
- Press p for primary.
- Press 3 for the partition number, depending the output of the partition table print.
- Press Enter two times.
- Press w to write the changes to the partition table.
- Restart the virtual machine.
- Run this command to verify that the changes were saved to the partition table and that the new partition has an 83 type:
# fdisk -l
- Run this command to convert the new partition to a physical volume:
# pvcreate /dev/sda3
- Run this command to extend the physical volume:
# vgextend VolGroup00 /dev/sda3
Note: To determine which volume group to extend, use the command vgdisplay.
- Run this command to verify how many physical extents are available to the Volume Group:
# vgdisplay VolGroup00 | grep "Free"
- Run the following command to extend the Logical Volume:
# lvextend -L+#G /dev/VolGroup00/LogVol00
Where # is the number of Free space in GB available as per the previous command.
Note: to determine which logical volume to extend, use the command lvdisplay.
- Run the following command to expand the ext3 filesystem online, inside of the Logical Volume:
# ext2online /dev/VolGroup00/LogVol00
Note: Use resize2fs instead of ext2online if it is not a Red Hat virtual machine.
- Run the following command to verify that the / filesystem has the new space available:
# df -h /
vSphere 5.1: vMotion without Shared Storage
Windows Server 2012 & Hyper-V 3.0 Live Migration
XenServer Storage Motion (block migration) on OpenStack
Hypervisor competition is really starting to heat up. VMware just released vSphere 5.1 and Microsoft has recently released Windows Server 2012 and the new version of Hyper-V. A significant new feature available now in Hyper-V / Windows 2012 is a new disk format VHDX, which has a maximum size of 64TB. With the new filesystem in Windows Server 2012 (ReFS) the maximum volume size increases to 256TB ( NTFS was limited to 16TB @ 4K cluster size). So how does vSphere 5 and 5.1 compare and what are the key considerations and gotchas? What are the implications for business critical applications? Read on to find out.
Before we get started I’d like to say this article isn’t going to cover performance of large volumes. But rather the argument for supporting larger than 2TB individual virtual disks and large volumes. There are many considerations around performance, and I will cover some of the implications when you start to scale up volume size, but for particular performance design considerations I’d like to recommend you read my article titled Storage Sizing Considerations when Virtualizing Business Critical Applications.
The Case for Larger than 2TB Virtual Disks
Recently I have been having an interesting debate with some of my VCDX peers on the merits and reasons for having larger than 2TB virtual disk support in vSphere. As of vSphere 5 VMware supports 64TB VMFS5 datastores, and 64TB Physical Mode (Pass-through) Raw Device Maps (RDM’s), but the largest single VMDK file supported on a VMFS5 volume is still 2TB-512b (hereon after referred to as 2TB). The same 2TB limit applies to virtual mode RDMs also. In this debate I’ve been suggesting that for now “most” applications can be supported with the 2TB virtual disk limit. If larger than 2TB volumes are required for a VM that is very easily accommodated with in guest volume managers and device concatenation of multiple 2TB disks, or using an alternative to VMFS. However realistically this can only go so far. I plan to cover both the pros and the cons as I see them.
Pros:
- Support for an individual VM with larger than 120TB storage requirements, which is the theoretical limit with 4 x vSCSI controllers, each with 15 disks (60 disks total) at the maximum size of 2TB each. You’ll find out why it’s a theoretical limit later.
- Easier to manage less devices and less volumes and space can potentially be more efficiently utilised.
- No need to use in guest volume managers for very large volumes.
- Easier to support very large individual files >2TB without the use of in guest volume managers.
- It could be argued that losing one 2TB device from a in guest managed volume has the same risk profile as losing a single large volume of the same size as in both cases the entire volume is potentially lost.
Cons:
- Larger individual devices and volumes take longer to backup and restore. This may require a major change in data protection architecture.
- Larger volumes will potentially take longer to replicate and recover in a DR scenario.
- The risk profile of losing a large volume or device is significantly higher than losing a smaller device or volume. Losing a single smaller device where no volume manager is being used results in only the small device having to be recovered instead of everything.
- Larger individual devices still have the same number of IO queues to the vSCSI controller which effectively limits their performance. This increases the risk of running out of performance before running out of capacity (until ultra low latency solid state flash storage is of massive capacity and abundantly available anyway).
- Significantly harder to take snapshots. A snapshot could still grow to be equally as large as the original virtual disk. This is probably one of the more significant reasons that VMware hasn’t yet introduced VMDK’s above 2TB.
- Significantly longer to check disk for integrity if there is any type of corruption, how will it be recovered if it’s very large?
- Impact on Storage vMotion times.
In my opinion the arguments are pretty even. But as I always err on the side of performance, and I think having more devices of a smaller size in a lot of cases is a better option as this gives you far more access to more queues and more parallel IO channels. However this is only relevant for some applications, mostly OLTP and messaging type applications. File servers, data warehousing, big data and the like may well benefit greatly from larger volume sizes, and it would make those applications significantly easier to manage. But the requirements will all be driven by the applications and at the moment I only see a very small minority of workloads require storage capacities that would justify very large individual SCSI devices and where the performance tradeoffs from an IO parallelism perspective are acceptable. Most of those corner cases have a suitable alternative for now (discussed below).
Options for Larger than 2TB Volumes
So if you’ve looked at the requirements for your application and you decide that you need a volume larger than 2TB, what are your options with vSphere 5.x?
- Using one or more VMFS volumes with virtual disks up to 2TB and in guest volume managers to concatenate them. Implications: The more devices the more storage IO queues and potentially the more performance. Oracle RAC vMotion Supported. Theoretically supports up to 120TB storage per VM.
- Physical Mode RDM – Support up to 64TB individual device, more than 3PB per VM.Implications: No Storage vMotion, No Hypervisor Snapshot Support, No Cloning, No vSphere API’s for Data Protection Support (vADP), No vCloud Director Support, No FT Support, No Oracle RAC vMotion Support, No Clustering vMotion Support.
- In Guest iSCSI – Supports up to 16TB or greater individual devices depending on iSCSI target. Implications: No Storage vMotion (of iSCSI devices), No Hypervisor Snapshot Support (of iSCSI devices), No Cloning (of iSCSI devices), No vSphere API’s for Data Protection Support (vADP) (of iSCSI devices), vCloud Director Supported, FT Supported, vMotion Supported, Clustering vMotion Support, higher CPU utilization.
- In Guest NFS – Supports very large volumes depending on the array. Implications: No Storage vMotion (of NFS devices), No Hypervisor Snapshot Support (of NFS devices), No Cloning (of NFS devices), No vSphere API’s for Data Protection Support (vADP) (of NFS devices), vCloud Director Supported, FT Supported, vMotion Supported, Oracle RAC vMotion Support, higher CPU utilization.
You can’t evaluate the alternatives in isolation and to be fair they are workarounds that you wouldn’t even have to consider if larger than 2TB VMDK’s were possible. Physical Mode RDM’s in particular have operational implications, especially as you can’t use hypervisor snapshots, cloning, and no backup API integration, just to name a few. So any alternative you choose needs to be thoroughly considered.
The Gotcha with VMFS
If you are going to have databases or systems with large disk footprints (and have multiple per host) you may need to modify the ESXi VMFS Heap Size by changing the advanced setting VMFS3.MaxHeapSizeMB. Review KB 1004424 and Jason Boche’s article Monster VMs & ESX(i) Heap Size: Trouble In Storage Paradise. Currently VMFS5 is limited to a maximum of25TB of virtual disks open per host (Yes per host). With a default setting allowing only 8TB of VMDK’s to be open per host. This means even if it is acceptable to you for a single VM to have multiple virtual disks of 2TB and using in guest volume managers you would not be able to configure or open more than 25TB total maximum on a single host (was 32TB with VMFS3). This is why the limit of 120TB per VM on VMFS is at this point purely theoretical.
If you want to work around this limitation you will need to adopt option 2, 3 or 4 above or use virtual mode RDMs. The reason is this limit is purely with VMFS and doesn’t impact RDM’s (physical or virtual) or in guest iSCSI or NFS.
[Updated 20/09/2012] A great example where it would be good to be able to support > 25TB VMDK’s per host and > 2TB per VMDK is where a customer has a requirement such as virtualizing 20 x 4TB File Servers. Each fileserver may not need much in the way or RAM or CPU, but does need a decent amount of storage. In theory these 20 VM’s could easily be consolidated on a single host (although wouldn’t be for availability requirements), but because the VMFS limitation this is not possible, and due to the limit of 2TB per VMDK limit you will require a minimum of 2 VMDK’s per VM. It may be more convenient to have a single 4TB VMDK for these types of servers. One option is to design for a consolidation ration of 5:1 and size the physical hosts accordingly, making sure to increase the default VMFS heap size. However this would introduce additional operational costs and effort. This brings us back to option 2, 3 and 4 above again. In this case vRDM may be a better option than pRDM even with the 2TB limit as it allows easy migration to VMFS / VMDK’s in the future. pRDM would have the advantage of reducing the number of LUNs in total required for the VM’s, which might be 60 LUNs in total, not taking into account other VM’s and LUNs in the cluster (which could bring them close to the 256 LUN limit per host), but with a tradeoff of a harder migration path in the future.
Final Word
Microsoft appears to have put the cat squarely among the pigeons in terms of large virtual disk storage support with their latest release of Windows 2012 and Hyper-V. In this respect VMware is indeed playing catch up. But are greater than 2TB virtual disks really required right now for most applications? In my opinion no. For the majority of applications the existing vSphere hypervisor can adequately cater for their size and performance needs. But this is only going to last so long. There are some good use cases documented in Cormac Hogan’s blog article How Much Storage Can I Present to a Virtual Machine.
Most applications in my experience, especially the performance and latency sensitive messaging and OLTP database applications would benefit more from a greater number of SCSI devices and queues. In their case supporting more than 256 datastores per host would be of benefit, especially if there are multiple of them all grouped in a cluster. The benefits of using VMFS and virtual disks are compelling and not being able to support very large virtual disks is definitely going to be a major problem in the future, considering VMFS5 already supports 64TB volumes. Especially considering the explosive growth of data. But do we want larger virtual disks and to sacrifice functionality, such as snapshots? I don’t think so. I hope that VMware will support larger virtual disks, even if they increase it up to 4TB or 16TB, and without sacrificing functionality. However in the meantime the alternatives such as RDMs and in guest storage access will fill the gap for some of the minority of workloads that need it, with the resulting trade offs in functionality. For those workloads where the workarounds are unacceptable they may not be virtualization candidates, at least on vSphere anyway, till some of these problems are solved.
Just because you can do something doesn’t mean you necessarily should. The back end array architecture needs to be considered and so does the data protection and disaster recovery protection aspects of the solution. It’s not good having a massive volume and a massive amount of storage per VM if you can’t protect that data and recovery it in a reasonable timeframe when required. I would like to know of your use cases that require greater than 2TB virtual disks and of your very large data Monster VM’s. Hopefully if there are enough customers that require larger than 2TB VMDK’s VMware will implement the necessary changes.
Here is what I’d like to see from VMware (In no particular order):
- Larger than 2TB VMDK Support
- More than 4 vSCSI Controllers per VM
- More than 256 SCSI Devices per Host
I would be very interested to get your feedback on this.