Planned Hybrid SSD/FC Disk Filesystems at CSIRO

Background
----------
CSIRO ASC currently use TP9700, TP9500 and IS220 disk on their Altix
4700.

The main DMF-managed filesystem is 6.2 TiB of TP9700 with a DCM
of 12 TiB IS220, with 18 million inodes and growing exponentially.
DMF holdings are 2 x 1011 TiB.

DMF also manages two 700 GiB administrative filesystems with 17 and
20 million inodes used for backups.  Together, the DMF holdings for
these two are 1 x 8.90 TiB.

With this configuration, this many inodes results in performance
problems for xfsdump and for some DMF operations, especially on
IS220 disk.

New configuration
-----------------
We have taken delivery of an IS4600 with
	- dual controllers
	- 6 x 73GB SSD
	- 74 x 450GB FC
	- 60 x 1TB SATA

The SATA will probably become our new DCM and the SSD and FC will be
our new main DMF-managed filesystem.

The planned use of the SSD was first suggested by David Honey of SGI,
and evolved as a result of experimentation by CSIRO.

Initially, the SSD was to be used to build an XVM filesystem,
which would have inodes and the XFS log (ie: journal) preallocated.
Then the filesystem will be grown to include the FC disk with the
mkfs.xfs "maxpct" option used to prevent future allocation of inodes
on the FC disk.  (Note that the mkfs "agcount" parameter was not
useful as xfs_growfs overrides it.)  We believe this would work,
at the cost of ensuring that the SSD was at least 1% of the total
filesystem which for us meant using all the SSD on just one filesystem.

This is wasteful of the SSD though, and Geoffrey Wehrman of SGI
Engineering looked into an alternative way of confining the inodes,
resulting in Bill's announcement here yesterday.  This new XFS "ibound"
mount option will be used to constrain the inodes to the SSD part of
the hybrid filesystem without relying on "maxpct", allowing us to use
SSD for all our busy filesystems' inodes, with some left over.

Expectations
------------
We expect that the increased speed of accessing inodes will benefit
(in decreasing order of importance to us):

	- xfsdump
	- dmfsfree
	- dmscanfs
	- dmaudit
	- dmmigrate
	- dmdaux

We don't have figures on the speedups, but we are assured that it will
be significant.

One downside of SSD is the reduced lifetime caused by heavy write
workloads.  We plan on configuring the SSD as RAID6 4+2P which will
allow two SSD modules to be down simultaneuosly without data loss.
Hopefully this is an acceptable risk.

We will evaluate the possibility of keeping the DMF databases on SSD,
and investigate whether the SMART software will adequately advise of
future SSD failures.

We hope to start installing this in the next month or two, and believe
that this is the first such configuration in the SGI world, outside
SGI's labs.

CSIRO would like to thank David and Geoffrey in particular, for their
assistance.

				Peter Edwards - CSIRO ASC - May 2010