text:
ZFS Fun
This tutorial is under a heavy revision to be switched from ZFS Fuse to ZFS on Linux.
ZFS features and limitations
ZFS offers an impressive amount of features even putting aside its hybrid nature (both a filesystem and a volume manager -- zvol) covered in detail on Wikipedia. One of the most fundamental points to keep in mind about ZFS is it targets a legendary reliability in terms of preserving data integrity. ZFS uses several techniques to detect and repair (self-healing) corrupted data. Simply speaking it makes an aggressive use of checksums and relies on data redundancy, the price to pay is a bit more CPU processing power. However, the Wikipedia article about ZFS also mention it is strongly discouraged to use ZFS over classic RAID arrays as it can not control the data redundancy, thus ruining most of its benefits.
In short, ZFS has the following features (not exhaustive):
- Storage pool dividable in one or more logical storage entities.
- Plenty of space:
- 256 zettabytes per storage pool (2^64 storages pools max in a system).
- 16 exabytes max for a single file
- 2^48 entries max per directory
- Virtual block-devices support support over a ZFS pool (zvol) - (extremely cool when jointly used over a RAID-Z volume)
- Read-only Snapshot support (it is possible to get a read-write copy of them, those are named clones)
- Encryption support (supported only at ZFS version 30 and upper, ZFS version 31 is shipped with Oracle Solaris 11 so that version is mandatory if you plan to encrypt your ZFS datasets/pools)
- Built-in RAID-5-like-over-steroid capabilities known as RAID-Z and RAID-6-like-over-steroid capabilities known as RAID-Z2. RAID-Z3 (triple parity) also exists.
- Copy-on-Write transactional filesystem
- Meta-attributes support (properties) allowing you to you easily drive the show like "That directory is encrypted", "that directory is limited to 5GiB", "That directory is exported via NFS" and so on. Depending on what you define, ZFS takes the appropriates actions!
- Dynamic striping to optimize data throughput
- Variable block length
- Data deduplication
- Automatic pool re-silvering
- Transparent data compression
- Transparent encryption (Solaris 11 and later only)
Most notable limitations are:
- Lack a features ZFS developers knows as "Block Pointer rewrite functionality" (planned to be developed), without it ZFS suffers of currently not being able to:
- Pool defragmentation (COW techniques used in ZFS mitigates the problem)
- Pool resizing
- Data compression (re-applying)
- Adding an additional device in a RAID-Z/Z2/Z3 pool to increase it size (however, it is possible to replace in sequence each one of the disks composing a RAID-Z/Z2/Z3)
- NOT A CLUSTERED FILESYSTEM like Lustre, GFS or OCFS2
- No data healing if used on a single device (corruption can still be detected), workaround if to force a data duplication on the drive
- No support of TRIMming (SSD devices)
ZFS on well known operating systems
Despite the source code of ZFS is open, its license (Sun CDDL) is incompatible with the license governing the Linux kernel (GNU GPL v2) thus preventing its direct integration. However a couple of ports exists, but suffers of maturity and lack of features. As of writing (February 2014) two known implementations exists:
- ZFS-fuse: a totally userland implementation relying on FUSE. This implementation can now be considered as defunct as of February 2014). The original site of ZFS FUSE seems to have disappeared nevertheless the source code is still available on http://freecode.com/projects/zfs-fuse. ZFS FUSE stalled at version 0.7.0 in 2011 and never really evolved since then.
- ZFS on Linux: a kernel mode implementation of ZFS in kernel mode which supports a lot of NFS features. The implementation is not as complete as it is under Solaris and its siblings like OpenIndiana (e.g. SMB integration is still missing, no encryption support...) but a lot of functionality is there. This is the implementation used for this article. As ZFS on Linux is an out-of-tree Linux kernel implementation, patches must be waited after each Linux kernel release. ZfsOnLinux currently supports zpools version 28.
- Oracle Solaris: remains the de facto reference platform for ZFS implementation: ZFS on this platform is now considered as mature and usable on production systems. Solaris 11 uses ZFS even for its "system" pool (aka rpool). A great advantage of this: it is now quite easy to revert the effect of a patch at the condition a snapshot has been taken just before applying it. In the "old good" times of Solaris 10 and before, reverting a patch was possible but could be tricky and complex when possible. ZFS is far from being new in Solaris as it takes its roots in 2005 to be, then, integrated in Solaris 10 6/06 introduced in June 2006.
- OpenIndiana: is based on the Illuminos kernel (a derivative of the now defunct OpenSolaris) which aims to provide absolute binary compatibility with Sun/Oracle Solaris. Worth mentioning that Solaris kernel and the Illumos kernel were both sharing the same code base, however, they now follows a different path since Oracle announced the discontinuation of OpenSolaris (August 13th 2010). Like Oracle Solaris, OpenIndiana uses ZFS for its system pool. The illumos kernel ZFS support lags a bit behind Oracle: it supports zpool version 28 where as Oracle Solaris 11 has zpool version 31 support, data encryption being supported at zpool version 30.
- FreeBSD: ZFS is present in FreeBSD since FreeBSD 7 (zpool version 6) and FreeBSD can boot on a ZFS volume (zfsboot). ZFS support has been vastly enhanced in FreeBSD 8.x (8.2 supports zpool version 15, version 8.3 supports version 28), FreeBSD 9 and FreeBSD 10 (both supports zpool version 28). ZFS in FreeBSD is now considered as fully functional and mature. FreeBSD derivatives such as the popular FreeNAS takes befenits of ZFS and integrated it in their tools. In the case of that latter, it have, for example, supports for zvol though its Web management interface (FreeNAS >= 8.0.1).
- NetBSD: ZFS has been started to be ported as a GSoC project in 2007 and is present in the NetBSD mainstream since 2009 (zpool version 13).
- OpenBSD: No ZFS support yet and not planned until Oracle changes some policies according to the project FAQ.
ZFS alternatives
- WAFL seems to have severe limitation [1] (document is not dated), also an interesting article lies here
- BTRFS is advancing every week but it still lacks such features like the capability of emulating a virtual block device over a storage pool (zvol) and built-in support for RAID-5/6 is not complete yet (cf. Btrfs mailing list). At date of writing, it is still experimental where as ZFS is used on big production servers.
- VxFS has also been targeted by comparisons like this one (a bit controversial). VxFS has been known in the industry since 1993 and is known for its legendary flexibility. Symantec acquired VxFS and proposed a basic version (no clustering for example) of it under the same Veritas Storage Foundation Basic
- An interesting discussion about modern filesystems can be found on OSNews.com
ZFS vs BTRFS at a glance
Some key features in no particular order of importance between ZFS and BTRFS:
Feature | ZFS | BTRFS | Remarks |
Transactional filesystem | YES | YES | |
Journaling | NO | YES | Not a design flaw, but ZFS is robust by design... See page 7 of "ZFS The last word on filesystems". |
Dividable pool of data storage | YES | YES | |
Read-only snapshot support | YES | YES | |
Writable snapshot support | YES | YES | |
Sending/Receiving a snapshot over the network | YES | YES | |
Rollback capabilities | YES | YES | While ZFS knows where and how to rollback the data (on-line), BTRFS requires a bit more work from the system administrator (off-line). |
Virtual block-device emulation | YES | NO | |
Data deduplication | YES | YES | Built-in in ZFS, third party tool (bedup) in BTRFS |
Data blocks reoptimization | NO | YES | ZFS is missing a "block pointer rewrite functionality", true on all known implementations so far. Not a major performance crippling however. BTRFS can do on-line data defragmentation. |
Built-in data redundancy support | YES | YES | ZFS has a sort of RAID-5/6 (but better! RAID-Z{1,2,3}) capability, BTRFS only fully supports data mirroring at this point, however some works remains to be done on parity bits handling by BTRFS. |
Management by attributes | YES | NO | Nearly everything touching ZFS management is related to attributes manipulation (quotas, sharing over NFS, encryption, compression...), BTRFS also retain the concept but it les less aggressively used. |
Production quality code | NO | NO | ZFS support in Linux is not considered as production quality (yet) although it is very robust. Several operating systems like Solaris/OpenIndiana have a production quality implementation, Solaris/OpenIndiana is now installed in ZFS datasets by defaults. |
Integrated within the Linux kernel tree | NO | YES | ZFS is released under the CDDL license... |
ZFS resource naming restrictions
Before going further, you must be aware of restrictions concerning the names you can use on a ZFS filesystem. The general rule is: you can can use all of the alphanumeric characters plus the following specials are allowed:
- Underscore (_)
- Hyphen (-)
- Colon (:)
- Period (.)
The name used to designate a ZFS pool has no particular restriction except:
- it can't use one the reserved words in particular:
- mirror
- raidz (raidz2, raidz3 and so on)
- spare
- cache
- log
- names must begin with an alphanumeric character (same for ZFS datasets).
Some ZFS concepts
Once again with no particular order of importance:
ZFS | What it is | Counterparts examples |
zpool | A group of one or many physical storage media (hard drive partition, file...). A zpool has to be divided in at least one ZFS dataset or at least one zvol to hold any data. Several zpools can coexists in a system at the condition they each hold a unique name. Also note that zpools can never be mounted, the only things that can are the ZFS datasets they hold. |
dataset | A logical subdivision of a zpool mounted in your host's VFS where your files and directories resides. Several uniquely named ZFS datasets can coexist in a single system at the conditions they each own a unique name within their zpool. |
snapshot | A read-only photo of a ZFS dataset state as is taken at a precise moment of time. ZFS has no way to cooperate on its own with applications that read and write data on ZFS datasets, if those latter still hold data at the moment the snapshot is taken, only what has been flushed will be included in the snapshot. Worth mentioning that snapshot do not take diskspace aside of sone metadata at the exact time they are created, they size will grow as more and data blocks (i.e. files) are deleted or changed on their corresponding live ZFS dataset. |
clone | What is is... A writable physical clone of snapshot |
zvol | An emulated block device whose data is hold behind the scene in the zpool the zvol has been created in. | No known equivalent even in BTRFS |
Your first contact with ZFS
- ZFS userland tools installed (package sys-fs/zfs)
- ZFS kernel modules built and installed (package sys-fs/zfs-kmod), there is a known issue with kernel 3.13 series see this thread on Funtoo's forum
- Disk size of 64 Mbytes as a bare minimum (128 Mbytes is the minimum size of a pool). Multiple disk will be simulated through the use of several raw images accessed via the Linux loopback devices.
- At least 512 MB of RAM
Once your have emerged sys-fs/zfs and sys-fs/zfs-kmod you have two options to start using ZFS at this point :
- Either you start /etc/init.d/zfs (will load all of the zfs kernel modules for you plus a couple of other things)
- Either you load the zfs kernel modules by hand (will load all of the zfs kernel modules for you)
So :
# rc-service zfs start
# modprobe zfs # lsmod | grep zfs zfs 874072 0 zunicode 328120 1 zfs zavl 12997 1 zfs zcommon 35739 1 zfs znvpair 48570 2 zfs,zcommon spl 58011 5 zfs,zavl,zunicode,zcommon,znvpair
Your first ZFS pool
To start with, four raw disks (2 GB each) are created:
# for i in 0 1 2 3; do dd if=/dev/zero of=/tmp/zfs-test-disk0${i}.img bs=2G count=1; done
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB) copied, 40.3722 s, 53.2 MB/s
Then let's see what loopback devices are in use and which is the first free:
# losetup -a # losetup -f /dev/loop0
In the above example nothing is used and the first available loopback device is /dev/loop0. Now associate all of the disks with a loopback device (/tmp/zfs-test-disk00.img -> /dev/loop/0, /tmp/zfs-test-disk01.img -> /dev/loop/1 and so on):
# for i in 0 1 2 3; do losetup /dev/loop${i} /tmp/zfs-test-disk0${i}.img; done # losetup -a /dev/loop0: [000c]:781455 (/tmp/zfs-test-disk00.img) /dev/loop1: [000c]:806903 (/tmp/zfs-test-disk01.img) /dev/loop2: [000c]:807274 (/tmp/zfs-test-disk02.img) /dev/loop3: [000c]:781298 (/tmp/zfs-test-disk03.img)
ZFS literature often names zpools "tank", this is not a requirement you can use whatever name of you choice (as we did here...)
Every story in ZFS takes its root with a the very first ZFS related command you will be in touch with: zpool. zpool as you might guessed manages all ZFS aspects in connection with the physical devices underlying your ZFS storage spaces and the very first task is to use this command to make what is called a pool (if you have used LVM before, volume groups can be seen as a counter part). Basically what you will do here is to tell ZFS to take a collection of physical storage stuff which can take several forms like a hard drive partition, a USB key partition or even a file and consider all of them as a single pool of storage (we will subdivide it in following paragraphs). No black magic here, ZFS will write some metadata on them behind the scene to be able to track which physical device belongs to what pool of storage.
# zpool create myfirstpool /dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3
And.. nothing! Nada! The command silently returned but it did something, let's check our the list of currently known pools:
Your first ZFS dataset
# zpool list
myfirstpool 7.94G 130K 7.94G 0% 1.00x ONLINE -
What does this mean? Several things: First, your zpool is here and has a size of, roughly, 8 Go minus some space eaten by some metadata. Second is is actually usable because the column HEALTH says ONLINE. Other columns are not meaningful for us for the moment just ignore them. If want more crusty details you can use the zpool command like this:
# zpool status
pool: myfirstpool
state: ONLINE
scan: none requested
myfirstpool ONLINE 0 0 0
loop0 ONLINE 0 0 0
loop1 ONLINE 0 0 0
loop2 ONLINE 0 0 0
loop3 ONLINE 0 0 0
Information is quite intuitive: your pool is seen as being usable (state is similar to HEALTH) and is composed of several devices each one listed as being in a healthy state ... at least for now because they will be salvaged for demonstration purpose in a later section. For your information the columns READ,WRITE and CKSUM list the number of operation failures on each of the devices respectfully:
- READ for reading failures. Having a non-zero value is not a good sign... the device is clunky and will soon fail.
- WRITE for writing failures. Having a non-zero value is not a good sign... the device is clunky and will soon fail.
- CKSUM for mismatch between the checksum of the data at the time is had been written and how it has been recomputed when read again (yes, ZFS uses checksums in a agressive manner). Having a non-zero value is not a good sign... corruption happened, ZFS will do its best to recover data by its own but this is definitely not a good sign of a healthy system.
Cool! So far so good you have a new 8 Gb usable brand new storage space on you system. Has been mounted somewhere?
# mount | grep myfirstpool
/myfirstpool on /myfirstpool type zfs (rw,xattr)
Remember the tables in the section above? A zpool in itself can never be mounted, never ever. It is just a container where ZFS datasets are created then mounted. So what happened here? Obscure black magic? No, of course not! Indeed a ZFS dataset named after the zpool's name should have been created automatically for us then mounted. Is is true? We will check this shortly. For the moment you will be introduced with the second command you will deal with when using ZFS : zfs. While the zpool command is used with anything related to zpools, the zfs is used to anything related to ZFS datasets (a ZFS dataset always resides in a zpool, always no exception on that).
zfs and zpool commands are the two only ones you will need to remember when dealing with ZFS.
So how can we check what ZFS datasets are currently known by the system? As you might already guessed like this:
# zfs list
myfirstpool 114K 7.81G 30K /myfirstpool
Tala! The mystery is busted! the zfs command tells us that not only a ZFS dataset named myfirstpool has been created but also it has been mounted in the system's VFS for us. If you check with the df command, you should also see something like this:
# df -h
Filesystem Size Used Avail Use% Mounted on
myfirstpool 7.9G 0 7.9G 0% /myfirstpool
The $100 question:"what to do with this band new ZFS /myfirstpool dataset ?". Copy some files on it of course! We used a Linux kernel source but you can of course use whatever you want:
# cp -a /usr/src/linux-3.13.5-gentoo /myfirstpool # ln -s /myfirstpool/linux-3.13.5-gentoo /myfirstpool/linux # ls -lR /myfirstpool /myfirstpool: total 3 lrwxrwxrwx 1 root root 32 Mar 2 14:02 linux -> /myfirstpool/linux-3.13.5-gentoo drwxr-xr-x 25 root root 50 Feb 27 20:35 linux-3.13.5-gentoo /myfirstpool/linux-3.13.5-gentoo: total 31689 -rw-r--r-- 1 root root 18693 Jan 19 21:40 COPYING -rw-r--r-- 1 root root 95579 Jan 19 21:40 CREDITS drwxr-xr-x 104 root root 250 Feb 26 07:39 Documentation -rw-r--r-- 1 root root 2536 Jan 19 21:40 Kbuild -rw-r--r-- 1 root root 277 Feb 26 07:39 Kconfig -rw-r--r-- 1 root root 268770 Jan 19 21:40 MAINTAINERS (...)
A ZFS dataset behaves like any other filesystem: you can create regular files, symbolic links, pipes, special devices nodes, etc. Nothing mystic here.
Now we have some data in the ZFS dataset let's see what various commands report:
# df -h
Filesystem Size Used Avail Use% Mounted on
myfirstpool 7.9G 850M 7.0G 11% /myfirstpool
# zfs list
myfirstpool 850M 6.98G 850M /myfirstpool
# zpool list
myfirstpool 7.94G 850M 7.11G 10% 1.00x ONLINE -
Notice the various sizes reported by zpool and zfs commands. In this case it is the same however it can differ, this is true especially with zpools mounted in RAID-Z.
Unmounting/remounting a ZFS dataset
Once again, remember that only ZFS datasets can be mounted inside your host's VFS, no exception on that okay? Zpools cannot be mounted, never, never, never... please pay attention to the terminology and keep things clear by not messing up with terms. We will introduce ZFS snapshots and ZFS clones in a later but those are ZFS datasets at the basis so they can also be mounted and unmounted.
If a ZFS dataset behaves just like any other filesystem, can we unmount it?
# umount /myfirstpool # mount | grep myfirstpool
No more /myfirstpool the line of sight! So yes, it is possible to unmount a ZFS dataset just like you would do with any other filesystem. Is the ZFS dataset still present on the system even it is unmounted? Let's check:
# zfs list
myfirstpool 850M 6.98G 850M /myfirstpool
Hopefully and obviously it is else ZFS would not be very useful. Your next concern would certainly be: "How can we remount it then?" Simple! Like this:
# zfs mount myfirstpool # mount | grep myfirstpool myfirstpool on /myfirstpool type zfs (rw,xattr)
The ZFS dataset is back! :-)
Your first contact with ZFS management by attributes or the end of /etc/fstab
At this point you might be curious about how the zfs command know what it has to mount and where is has to mount it. You might be familiar with the following syntax of the mount command that, behind the scenes, scans the file /etc/fstab and mount the specified entry:
# mount /boot
Does /etc/fstab contains something related to our ZFS dataset?
# cat /etc/fstab | grep myfirstpool
Doh!!!... Obvisouly nothing there. Another mystery? Sure not! The answer lies in a extremely powerful feature of ZFS: the attributes. Simply speaking: an attribute is named property of a ZFS dataset that holds a value. Attributes govern various aspects of how the datasets are managed like: "Is the data has to be compressed?", "Is the data has to be encrypted?", "Is the data has to be exposed to the rest of the world by NFS or SMB/Samba?" and of course... '"Where the dataset has to be mounted?". The answer to that latter question can be tell by the following command:
# zfs get mountpoint myfirstpool
myfirstpool mountpoint /myfirstpool default
Bingo! When you remounted the dataset just some paragraphs ago, ZFS automatically inspected the mountpoint attribute and saw this dataset has to be mounted in the directory /myfirstpool.
A step forward with ZFS datasets
So far you were given a quick tour of what ZFS can do for you and it is very important at this point to distinguish a zpool from a ZFS dataset and to call a dataset for what it is (a dataset) and not for what is is not (a zpool). It is a bit confusing and an editorial choice to have choosen a confusing name just to make you familiar with the one and the other.
Creating datasets
Obviously it is possible to have more than one ZFS dataset within a single zpool. Quizz: what command would you use to subdivide a zpool in datasets? zfs or zpool? Stops reading for two seconds and try to figure out this little question. Frankly.
Answer is... zfs! Although you want to operate on the zpool to logically subdivide it in several datasets, you manage datasets at the end thus you will use the zfs command. It is not always easy at the beginning, do not be too worry you will soon get the habit when to use one or the other. Creating a dataset in a zpool is easy: just give to the zfs command the name of the pool you want to divide and the name of the dataset you want to create in it. So let's create three datasets named myfirstDS, mysecondDS and mythirdDS in myfirstpool(observe how we use the zpool and datasets' names) :
# zfs create myfirstpool/myfirstDS # zfs create myfirstpool/mysecondDS # zfs create myfirstpool/mythirdDS
What happened? Let's check :
# zfs list
myfirstpool 850M 6.98G 850M /myfirstpool
myfirstpool/myfirstDS 30K 6.98G 30K /myfirstpool/myfirstDS
myfirstpool/mysecondDS 30K 6.98G 30K /myfirstpool/mysecondDS
myfirstpool/mythirdDS 30K 6.98G 30K /myfirstpool/mythirdDS
Obviously we have there what we asked. Moreover if we inspect the contents of /myfirstpool:
# ls -l /myfirstpool total 8 lrwxrwxrwx 1 root root 32 Mar 2 14:02 linux -> /myfirstpool/linux-3.13.5-gentoo drwxr-xr-x 25 root root 50 Feb 27 20:35 linux-3.13.5-gentoo drwxr-xr-x 2 root root 2 Mar 2 15:26 myfirstDS drwxr-xr-x 2 root root 2 Mar 2 15:26 mysecondDS drwxr-xr-x 2 root root 2 Mar 2 15:26 mythirdDS </command> ZFS datasets are appearing just as if they were regular directories. Are they? Try to remove one of those: <console> # rmdir /myfirstpool/myfirstDS rmdir: failed to remove `/myfirstpool/myfirstDS': Device or resource busy
This behavior is absolutely normal. Remember: ZFS datasets are mounted when created.
# mount
myfirstpool on /myfirstpool type zfs (rw,xattr)
myfirstpool/myfirstDS on /myfirstpool/myfirstDS type zfs (rw,xattr)
myfirstpool/mysecondDS on /myfirstpool/mysecondDS type zfs (rw,xattr)
myfirstpool/mythirdDS on /myfirstpool/mythirdDS type zfs (rw,xattr)
As we did before, we can copy some files in the newly created datasets just like they were regular directories:
# cp -a /usr/portage /myfirstpool/mythirdDS # ls -l /myfirstpool/mythirdDS/* total 697 drwxr-xr-x 48 root root 49 Aug 18 2013 app-accessibility drwxr-xr-x 238 root root 239 Jan 10 06:22 app-admin drwxr-xr-x 4 root root 5 Dec 28 08:54 app-antivirus drwxr-xr-x 100 root root 101 Feb 26 07:19 app-arch drwxr-xr-x 42 root root 43 Nov 26 21:24 app-backup drwxr-xr-x 34 root root 35 Aug 18 2013 app-benchmarks drwxr-xr-x 66 root root 67 Oct 16 06:39 app-cdr(...)
Nothing really too exciting here, we have file in mythirdDS. A bit more interesting output:
# zfs list
myfirstpool 1.81G 6.00G 850M /myfirstpool
myfirstpool/myfirstDS 30K 6.00G 30K /myfirstpool/myfirstDS
myfirstpool/mysecondDS 30K 6.00G 30K /myfirstpool/mysecondDS
myfirstpool/mythirdDS 1002M 6.00G 1002M /myfirstpool/mythirdDS
# df -h
Filesystem Size Used Avail Use% Mounted on
myfirstpool 6.9G 850M 6.1G 13% /myfirstpool
myfirstpool/myfirstDS 6.1G 0 6.1G 0% /myfirstpool/myfirstDS
myfirstpool/mysecondDS 6.1G 0 6.1G 0% /myfirstpool/mysecondDS
myfirstpool/mythirdDS 7.0G 1002M 6.1G 15% /myfirstpool/mythirdDS
Noticed the size given for the 'AVAIL' column? At the very beginning of this tutorial we had slightly less than 8 Gb of available space, it now has a value of roughly 6 Gb. The datasets are just a subdivision of the zpool, they compete with each others for using the available storage within the zpool, no miracle here. To what limit? The pool itself as we never imposed a quota on datasets. Hopefully df and zfs list gives a coherent result.
Second contact with attributes: quota management
Remember how painful is the quota management under Linux? Now you can say goodbye to setquota, edquota and other quotacheck commands, ZFS handle of of this in the snap of a finger! Guess with what? An ZFS dataset attribute of course! ;-) Just to make you drool here is how a 2Gb limit can be put on myfirstpool/mythirdDS :
# zfs set quota=2G myfirstpool/mythirdDS
The zfs command is bit silent however if we check we can see that myfirstpool/mythirdDS is now capped to 2 Gb (forget 'REFER' for the moment): around 1 Gb of data has been copied in this dataset thus leaving a big 1 Gb of available space.
# zfs list
myfirstpool 1.81G 6.00G 850M /myfirstpool
myfirstpool/myfirstDS 30K 6.00G 30K /myfirstpool/myfirstDS
myfirstpool/mysecondDS 30K 6.00G 30K /myfirstpool/mysecondDS
myfirstpool/mythirdDS 1002M 1.02G 1002M /myfirstpool/mythirdDS
Destroying datasets
We have three datasets, but the third is pretty useless and contains a lot of garbage. Is it possible to remove it with a simple rm -rf? Let's try:
# rm -rf /myfirstpool/mythirdDS rm: cannot remove `/myfirstpool/mythirdDS': Device or resource busy
This is perfectly normal, remember that datasets are special entities that requires special care and they are not deletable through regular shell commands. However it is possible to destroy them and here again, the zfs command comes at our rescue:
# zfs destroy myfirstpool/mythirdDS # zfs list NAME USED AVAIL REFER MOUNTPOINT myfirstpool 444M 7.38G 444M /myfirstpool myfirstpool/myfirstDS 21K 7.38G 21K /myfirstpool/myfirstDS myfirstpool/mysecondDS 21K 7.38G 21K /myfirstpool/mysecondDS
Et voila! No more third dataset. :)
A bit more subtle case: let's mythirdDS and put another nested one in it then try to destroy mythirdDS again:
# zfs create myfirstpool/mythirdDS # zfs create myfirstpool/mythirdDS/nestedSD # zfs list NAME USED AVAIL REFER MOUNTPOINT myfirstpool 444M 7.38G 444M /myfirstpool myfirstpool/myfirstDS 21K 7.38G 21K /myfirstpool/myfirstDS myfirstpool/mysecondDS 21K 7.38G 21K /myfirstpool/mysecondDS myfirstpool/mythirdDS 42K 7.38G 21K /myfirstpool/mythirdDS myfirstpool/mythirdDS/nestedDS 21K 7.38G 21K /myfirstpool/mythirdDS/nestedDS # zfs destroy myfirstpool/mythirdDS cannot destroy 'myfirstpool/mythirdDS': filesystem has children use '-r' to destroy the following datasets: myfirstpool/mythirdDS/nestedDS
zfs tells us it has found some others datasets located in mythirdDS and, thus, is unable to delete it without you consent to make a recursive destruction (-r parameter). Before trying to destroy the dataset again let's create some more nested datasets plus a couple of directories inside mythirdDS:
# zfs create myfirstpool/mythirdDS/nestedSD # zfs create myfirstpool/mythirdDS/nestedSD2 # zfs create myfirstpool/mythirdDS/nestedSD3 # mkdir /myfirstpool/mythirdDS/dir1 # mkdir /myfirstpool/mythirdDS/dir2 # mkdir /myfirstpool/mythirdDS/dir3 # zfs list NAME USED AVAIL REFER MOUNTPOINT myfirstpool 444M 7.38G 444M /myfirstpool myfirstpool/myfirstDS 21K 7.38G 21K /myfirstpool/myfirstDS myfirstpool/mysecondDS 21K 7.38G 21K /myfirstpool/mysecondDS myfirstpool/mythirdDS 84K 7.38G 21K /myfirstpool/mythirdDS myfirstpool/mythirdDS/mynestedDS 21K 7.38G 21K /myfirstpool/mythirdDS/mynestedDS myfirstpool/mythirdDS/mynestedDS2 21K 7.38G 21K /myfirstpool/mythirdDS/mynestedDS2 myfirstpool/mythirdDS/mynestedDS3 21K 7.38G 21K /myfirstpool/mythirdDS/mynestedDS3 # zfs destroy -r myfirstpool/mythirdDS
Now what happens if we try to destroy mythird again this time with '-r'?
# zfs destroy -r myfirstpool/mythirdDS cannot destroy 'myfirstpool/mythirdDS/mynestedDS': dataset is busy
This is not as exactly normal as it should and seems to be a bug in zfs-fuse, the expected behavior is to automatically unmount any dataset contained inside mythirdDS then destroy it including mythirdDS itself. The same kind of operation on a Solaris machine with a similar dataset structure gives:
# zfs list NAME USED AVAIL REFER MOUNTPOINT .... rpool1/swap 4.04G 23.2G 123M - testpool/test 55.4K 3.76T 55.4K /testpool/test testpool/test/ds1 44.9K 3.76T 44.9K /testpool/test/ds1 testpool/test/ds2 44.9K 3.76T 44.9K /testpool/test/ds2 testpool/test/ds3 44.9K 3.76T 44.9K /testpool/test/ds3 testpool/test2 44.9K 3.76T 44.9K /testpool/test2 # mkdir /testpool/test/dir1 # mkdir /testpool/test/dir2 # mkdir /testpool/test/dir1 # zfs destroy -r testpool/test # zfs list NAME USED AVAIL REFER MOUNTPOINT .... rpool1/swap 4.04G 23.2G 123M - testpool/test2 44.9K 3.76T 44.9K /testpool/test2
To go back on ZFS Fuse, just do a few attempts and mythirdDS should vanish (you may also have to do an explicit zfs destroy mythirdDS at the end).
Snapshotting and rolling back a dataset
This is, by far, one of the coolest feature of ZFS: you can litterally take a photograph of a dataset, do whatever you want with the dataset then restore it in the exact same state just as if nothing had ever happened in the middle. To start with, let's copy some files in mysecondDS:
# cp -a /usr/portage /myfirstpool/mysecondDS # ls /myfirstpool/mysecondDS/portage total 200 drwxr-xr-x 45 root root 46 Aug 31 07:37 app-accessibility drwxr-xr-x 202 root root 203 Sep 2 07:21 app-admin drwxr-xr-x 3 root root 4 Aug 18 18:13 app-antivirus drwxr-xr-x 93 root root 94 Aug 18 18:13 app-arch ... drwxr-xr-x 57 root root 58 Aug 22 08:56 x11-wm drwxr-xr-x 16 root root 17 Aug 18 18:13 xfce-base drwxr-xr-x 54 root root 55 Aug 18 18:13 xfce-extra
Now, let's take a snapshot of mysecondDS. Because we manipulate a dataset and not the pool, we rely on the zfs command:
# zfs snapshot myfirstpool/mysecondDS@Charlie
The syntax is always pool/dataset@snapshot-name, the name of the snapshot is left at your discretion however you must use an at sign (@) to separate the snapshot name from the rest of the path.
After running that command,
# ls -la /myfirstpool/mysecondDS total 9 drwxr-xr-x 3 root root 3 Sep 5 16:49 . drwxr-xr-x 6 root root 6 Sep 5 15:43 .. drwxr-xr-x 164 root root 169 Aug 18 18:25 portage
You were not thinking you would see something like @Charlie or Charlie lying in /myfirstpool/mysecondDS were you? Of course not, this is obvious ;-) Can zfs be of any help this time? It has rescued us several times in the past:
# zfs list NAME USED AVAIL REFER MOUNTPOINT myfirstpool 2.27G 5.54G 444M /myfirstpool myfirstpool/myfirstDS 21K 5.54G 21K /myfirstpool/myfirstDS myfirstpool/mysecondDS 1.84G 5.54G 1.84G /myfirstpool/mysecondDS #
So where the heck is Charlie? And how on earth can we use it if *nothing* is visible to us. Again the answer is: zfs! This time we invoke it with the -t parameter set to 'all' meaning "list all dataset including snapshots":
# zfs list NAME USED AVAIL REFER MOUNTPOINT myfirstpool 2.27G 5.54G 444M /myfirstpool myfirstpool/myfirstDS 21K 5.54G 21K /myfirstpool/myfirstDS myfirstpool/mysecondDS 1.84G 5.54G 1.84G /myfirstpool/mysecondDS myfirstpool/mysecondDS@Charlie 37K - 1.84G - #
Notice that Charlie is not mounted and although mysecondDS holds near 2GB of data, Charlie takes only a couple of kilobytes in the dataset. This is the consequence of ZFS being a Copy-on-write filesystem, duplicating all of the data blocks is not required. They will be duplicated only when needed: when ZFS sense a change in a data block, it will create a copy of it thus leaving intact the datablock pointed by a snapshot. At the time they are taken, snapshots occupy very little space in the datasets however as the time goes on they tend to "stick"more and more data blocks to be in use. It is wise to delete snapshots when become not needed anymore.
OpenIndiana and Oracle Solaris supports an interesting feature not available in ZFS Fuse: a kind of secret door in the form of a virtual directory named .zfs (notice the dot ahead). "secret door" because it is really secret! You cannot see it even with ls -la, however .zfs is present in just any of your datasets and holds some very interesting clues:
# zfs list -t all ... testpool/test2 205K 3.76T 70.3K /testpool/test2 testpool/test2@snap1 0 - 70.3K - # cd /testpool/test2 # ls -la total 22 drwxr-xr-x 11 root root 11 2011-09-05 17:34 . drwxr-xr-x 6 root root 6 2011-09-05 16:13 .. drwxr-xr-x 2 root root 2 2011-09-05 17:34 .sometest drwxr-xr-x 2 root root 2 2011-09-05 17:34 .xyz drwxr-xr-x 2 root root 2 2011-09-05 16:13 dir1 drwxr-xr-x 2 root root 2 2011-09-05 16:13 dir2 ... # cd /testpool/test2/.zfs # pwd /testpool/test2/.zfs # ls -l ls -l total 2 dr-xr-xr-x 2 root root 2 2011-09-05 16:13 shares dr-xr-xr-x 3 root root 3 2011-09-05 17:19 snapshot # cd snapshot # ls -l total 2 drwxr-xr-x 9 root root 9 2011-09-05 17:19 snap1 # cd snap1 # ls -l total 22 drwxr-xr-x 11 root root 11 2011-09-05 17:34 . drwxr-xr-x 6 root root 6 2011-09-05 16:13 .. drwxr-xr-x 2 root root 2 2011-09-05 17:34 .sometest drwxr-xr-x 2 root root 2 2011-09-05 17:34 .xyz drwxr-xr-x 2 root root 2 2011-09-05 16:13 dir1 drwxr-xr-x 2 root root 2 2011-09-05 16:13 dir2 ...
Despite you cannot change the snapshot contents, you can access it without having to roll it back to examine its contents. Extremely nifty design choice from the ZFS designers!
Now we have found Charlie, let's do some changes in the mysecondDS:
# rm -rf /myfirstpool/mysecondDS/portage # echo "Hello, world" > /myfirstpool/mysecondDS/hello.txt # ls -l /myfirstpool/mysecondDS total 1 -rw-r--r-- 1 root root 13 Sep 5 18:07 hello.txt # cat /myfirstpool/mysecondDS/hello.txt Hello, world
Whooops...removing portage was not the best idea to have and we do not bother about hello.txt. We will have to move back at checkpoint Charlie!
# zfs rollback myfirstpool/mysecondDS@Charlie # ls -l /myfirstpool/mysecondDS total 6 drwxr-xr-x 164 root root 169 Aug 18 18:25 portage
Again, ZFS handled everything for you and you now have the contents of mysecondDS exactly as it was at the time the snapshot Charlie was taken. Not more complicated than that. Hang on you hat, we have not finished.
Dealing with several snapshots (time-traveling machine)
So far we only used a single snapshot just to keep things simple. However a dataset can hold several snapshots and moreover you can do a delta between two snapshots and nothing is really much more complicated than you have seen so far.
Let's consider myfirstDS this time. This dataset should be empty as we did nothing in it so far:
# ls -la /myfirstpool/myfirstDS total 3 drwxr-xr-x 2 root root 2 Sep 4 23:34 . drwxr-xr-x 6 root root 6 Sep 5 15:43 ..
Now generate some contents, take a snapshot (snapshot-1), add more content, take a snapshot again (snapshot-2), do some more modifications and take a third snapshot (snapshot-3):
# echo "Hello, world" > /myfirstpool/myfirstDS/hello.txt # cp /usr/src/linux-3.1-rc4.tar.bz2 /myfirstpool/myfirstDS # ls -l /myfirstpool/myfirstDS # ls -l /myfirstpool/myfirstDS total 75580 -rw-r--r-- 1 root root 13 Sep 5 22:38 hello.txt -rw-r--r-- 1 root root 77220912 Sep 5 22:38 linux-3.1-rc4.tar.bz2 # zfs snapshot myfirstpool/myfirstDS@snapshot-1 # echo "Goodbye, world" > /myfirstpool/myfirstDS/goodbye.txt # echo "Are you there?" >> /myfirstpool/myfirstDS/hello.txt # cp /usr/src/linux-3.0.tar.bz2 /myfirstpool/myfirstDS # rm /myfirstpool/myfirstDS/linux-3.1-rc4.tar.bz2 # zfs snapshot myfirstpool/myfirstDS@snapshot-2 # echo "Still there?" >> /myfirstpool/myfirstDS/goodbye.txt # rm /myfirstpool/myfirstDS/hello.txt # cp /proc/config.gz /myfirstpool/myfirstDS # zfs snapshot myfirstpool/myfirstDS@snapshot-3 # zfs list -t all # zfs list -t all NAME USED AVAIL REFER MOUNTPOINT myfirstpool 2.41G 5.40G 444M /myfirstpool myfirstpool/myfirstDS 147M 5.40G 73.3M /myfirstpool/myfirstDS myfirstpool/myfirstDS@snapshot-1 73.8M - 73.8M - myfirstpool/myfirstDS@snapshot-2 20K - 73.3M - myfirstpool/myfirstDS@snapshot-3 0 - 73.3M -
Wow, nice demonstration on how a Copy-on-Write filesystem like ZFS works: what do we observe? First it is quite obvious to see that snapshot-1 is quite big. Is is possible that having a so big snapshot to be the consequence of removing /myfirstDS/linux-3.1-rc4.tar.bz2? Absolutely. Remember that a snapshot is a photograph of what a dataset contains at a given time, deleted information and unmodified original information is retained by the snapshot even you delete it from the dataset or bring in some changes to it. If you look again at the command history between snapshot-2 and snapshot-3, you will notice that we removed a small file and changed another small file a bit thus having a little information delta between what the dataset content at this time and what it also actually contains leading to a very small snapshot at the end. The third dataset is the exact copy of what the current dataset contains thus its size is very close to zero (truncated to zero on what you see).
$100 question: "How can I see what changed between snapshots?". Answer: yes, you can! Nuance is: ZFS Fuse does not support it yet :( Nevertheless here is what snapshots diffing looks like on an OpenIndiana/Solaris machine:
# zfs create testpool/test2 # cd /testpool/test2 # wget http://www.kernel.org/pub/linux/kernel/v3.0/testing/patch-3.1-rc4.bz2 # echo "Hello,world" > hello.txt # zfs snapshot testpool/test2@s1 # rm patch-3.1-rc4.bz2 # echo 'Goodbye!' > goodbye.txt # echo 'Still there?' >> hello.txt # zfs snapshot testpool/test2@s2 # echo 'Hello, again' >> hello.txt # ln -s goodbye.txt goodbye2.txt # mv hello.txt hello-new.txt # zfs snapshot testpool/test2@s3 # zfs list -t all | grep test2 testpool/test2 8.49M 3.76T 47.9K /testpool/test2 testpool/test2@s1 8.41M - 8.42M - testpool/test2@s2 29.2K - 46.4K - testpool/test2@s3 0 - 47.9K - # zfs diff testpool/test2@s1 testpool/test2@s2 M /testpool/test2/ - /testpool/test2/patch-3.1-rc4.bz2 M /testpool/test2/hello.txt + /testpool/test2/goodbye.txt # zfs diff testpool/test2@s2 testpool/test2@s3 M /testpool/test2/ R /testpool/test2/hello.txt -> /testpool/test2/hello-new.txt + /testpool/test2/goodbye2.txt # zfs diff testpool/test2@s1 testpool/test2@s3 M /testpool/test2/ - /testpool/test2/patch-3.1-rc4.bz2 R /testpool/test2/hello.txt -> /testpool/test2/hello-new.txt + /testpool/test2/goodbye.txt + /testpool/test2/goodbye2.txt # zfs diff testpool/test2@s3 san/test2@s1 Unable to obtain diffs: Not an earlier snapshot from the same fs
Where M,R,+,- stands for:
- M: item has been modified
- R: item has been renamed
- +: item has been added
- -: item has been removed
Observe the output of each diff and draw you own conclusion on what we did at each step and what appears in the diff. It is not possible to get a detailed diff similar to what Git and others gives but you have the big picture of what changed between snapshots.
If ZFS-Fuse does not implements (yet) a snapshot diffing capability, it can deal with several snapshots and is able to jump across several steps backwards. Suppose we want myfirstDS to go back exactly is was when we took the dataset photograph named snapshot-1:
# zfs rollback myfirstpool/myfirstDS@snapshot-1 cannot rollback to 'myfirstpool/myfirstDS@snapshot-1': more recent snapshots exist use '-r' to force deletion of the following snapshots: myfirstpool/myfirstDS@snapshot-3 myfirstpool/myfirstDS@snapshot-2
This is not a bug, this is absolutely normal. The zfs command asks you to give it the explicit permission to remove the two others snapshots as they becomes useless (restoring them would be an absolute no sense) once snapshot-1 is restored. Second attempt:
# zfs rollback myfirstpool/myfirstDS@snapshot-1 # ls -l /myfirstpool/myfirstDS total 75580 -rw-r--r-- 1 root root 13 Sep 5 22:38 hello.txt -rw-r--r-- 1 root root 77220912 Sep 5 22:38 linux-3.1-rc4.tar.bz2 # zfs list -t all NAME USED AVAIL REFER MOUNTPOINT myfirstpool 2.34G 5.47G 444M /myfirstpool myfirstpool/myfirstDS 73.8M 5.47G 73.8M /myfirstpool/myfirstDS myfirstpool/myfirstDS@snapshot-1 0 - 73.8M - myfirstpool/mysecondDS 1.84G 5.47G 1.84G /myfirstpool/mysecondDS myfirstpool/mysecondDS@snapshot1 37K - 1.84G -
myfirstDS effectively returned to its state when snapshot-2 was taken and the snapshots snapshot-2 and snapshot-3 vanished.
You can leap several steps backward at the cost of loosing your subsequent modifications forever.
Streaming datasets over the network
Nothing in an infrastructure is as much critical as having reliable backups of the data used by an organization. Whereas a server can be remounted from scratch, the data it contains is very likely to be lost forever whenever a disaster occurs. Of course, as the data is the blood of an organization business processes, its integrity and confidentiality must be preserved in all cases.
You find ZFS snaphots useful? Well, you have seen just a small part of their potential. As a snapshot is a photograph of what a dataset contains frozen in the time, snapshots can be seen as being no more than a data backup. Like any backup, they must not stay on the local machine but must be put elsewhere and the common good sense tells to keep backups in a safe place, making them travel through a secure channel. By "secure channel" we intend something like a trusted person in your organization whose job consists of bringing a box of tapes off-site in a secure location but we also intend a secure communication channel like an SSH tunnel over two hosts without any human intervention.
ZSH designers had the same vision and made possible for a dataset to be able to be sent over a network. How is that possible? Simple: the process involves two peers who can use through a communication channel like the one established by netcat (OpenSSH supports a similar functionality but with an encrypted communication channel). For the sake of the demonstration, we will use two Solaris boxes at each end-point.
How stream some ZFS bits over the network? Here again, zfs is the answer. A nifty move from the designers was to use stdin and stdout as transmission/reception channels thus allowing great a flexibility in processing the ZFS stream. You can envisage, for instance, to compress your stream then crypt it then encode it in base64 then sign it and so on. It sounds a bit overkill but it is possible and in the general case you can use any tool that swallow the data from stdin and spit it through stdout in your plumbing.
The rest of this section has been done entirely on two Solaris 11 machines.
1. Sender side:
# zfs create testpool2/zfsstreamtest # echo 'Hello, world!' > /testpool2/zfsstreamtest/hello.txt # echo 'Goodbye, world' > /testpool2/zfsstreamtest/goodbye.txt # zfs snapshot zfs testpool2/zfsstreamtest@s1 # zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT testpool2/zfsstreamtest@s1 0 - 32K -
2. Receiver side (the dataset zfs-stream-test will be created and should not be present):
# nc -l -p 7000 | zfs receive testpool/zfs-stream-test
At this point the receiver is waiting after some data.
3. Sender side:
# zfs send testpool2/zfsstreamtest@s1 | nc 192.168.aaa.bbb.ccc 7000
4. Receiver side:
# zfs list -t snapshot NAME USED AVAIL REFER ... testpool2/zfs-stream-test@s1 0 - 46.4K -
Note that we did not set an explicit snapshot name in the second step but it could have been possible to choose anything else but the default which is the name of the snapshot sent over the network. In that case the dataset which will contain the snapshot needs to be created first:
# nc -l -p 7000 | zfs receive testpool/zfs-stream-test@mysnapshot01
Once received you would get:
# zfs list -t snapshot NAME USED AVAIL REFER ... testpool2/zfs-stream-test@mysnapshot01 0 - 46.4K -
5. Just for the sake of the curiosity let's do a rollback on the receiver side:
# zfs rollback testpool2/zfsstreamtest@s1 # ls -l /testpool2/zfs-stream-test total 2 -rw-r--r-- 1 root root 15 2011-09-06 23:54 goodbye.txt -rw-r--r-- 1 root root 13 2011-09-06 23:53 hello.txt # cat /testpool2/zfs-stream-test/hello.txt Hello, world
Because ZFS streaming operates using the starnd input and output (stdin / stdout) you can build a bit more complex pipeline like:
# zfs send testpool2/zfsstreamtest@s1 | gzip | nc 192.168.aaa.bbb.ccc 7000
The above example was using two hosts but a simpler setup is also possible: you are not required to send you data over the network with netcat, you can store it to a regular file then mail it or store it on a USB key. By the way: we have not finished! We took only a simple case here: it is absolutely possible to do the exact same operation with the difference between snapshots (incremental). Just like an incremental backup takes only what has changed, ZFS can determine the difference between two snapshots and streaming instead of streaming a snapshot taken at whole. Although ZFS can detect and act on differentials, it does not operate (yet) at the block level: if only a few bytes of a very big file have changed, the whole file will be taken into consideration (operating at data block level is possible with some tools like the well-known rsync).
Consider the following:
- A dataset snapshot (S1) contains two files:
- A bit later some files (named C, D and E) are added to the dataset and another snapshot is (S2) taken. S2 contains:
With a full transfer of S2 A,B,C,D and E would be streamed whereas an incremental transfert (S2-S1), zfs would only process C, D and E. The next $100 question:"How can we stream a difference of snapshot? zfs again?" Yes! This time with a subtle difference: a special option specified on the command line telling it must use a difference rather than a full snapshot. Assuming a few more files are added in testpool2/zfsstreamtest dataset and a snapshot (s2) is has been taken, the delta between s2 and s1 (s2-s1) giving s3 can be send like this (on the receiver side the same as shown above is used, nothing special is required alos notice the presence of the -i option):
- Sender:
# zfs send -i testpool2/zfsstreamtest@s1 testpool2/zfsstreamtest@s2 | nc 192.168.aaa.bbb.ccc 7000
- Receiver:
# nc -l -p 7000 | zfs receive testpool/zfs-stream-test # zfs list -t snapshot testpool/zfs-stream-test@s1 28.4K - 46.4K - testpool/zfs-stream-test@s2 0 - 47.1K -
Note that although we did not specified any snapshot name to use on the receiver side, ZFS used by default the name of the second snapshot involved in the delta (s2 here).
$200 question: suppose we delete all of the received snapshots so far on the receiver side and we try to send the difference between s2 and s1, what would happen? ZFS will protest on the receiver side although no error message will be visible on the sender side:
cannot receive incremental stream: destination testpool/zfs-stream-test has been modified since most recent snapshot
It is even worse if we remove the dataset used to receive the data:
cannot receive incremental stream: destination 'testpool/zfs-stream-test' does not exist
ZFS streaming over a network has no underlying protocol, therefore the sender just assumes the data has been successfully received and processed. It does not care whether a processing error occurs.
Govern a dataset by attributes
So far, most of a filesystem capabilities were driven by separate and scarced command line line tools (e.g. tune2fs, edquota, rquota, quotacheck...) which all have their own ways to handle tasks and can go through tricky ways sometimes especially the quota-related management utilities. Moreover, there was no easy way to handle a limitations on a directory rather than putting it a a dedicated partition or logical volume implying downtimes when additional space was to be added. Quota management is however one of the many facets disk space management includes.
In the ZFS world, many aspects are now managed by simply setting/clearing a property attached to a ZFS dataset through the now so well-known command zfs.You can, for example:
- put a size limit on a dataset
- reserve a space for dataset (that space is guaranteed to be available in the future although not being allocated at the time the reservation is made)
- control if new files are encrypted and/or compressed
- define a quota per user or group of users
- control checksum usage => never turn that property off unless having very good reasons you are likely to never have (no checksums = no silent data corruption detection)
- share a dataset by NFS/CIFS
- control automatic data deduplication
Not all of a dataset properties are settable, some of them are set and managed by the operating system in the background for you and thus cannot be modified.
Solaris/OpenIndiana users: ZFS has a tight integration with the NFS/CIFS server, thus it is possible to share a zfs dataset by setting adequate attributes. ZFS on Linux (native kernel mode port) also has a tight integration with the built-in Linux NFS server, the same for ZFS fuse although still experimental. Under FreeBSD ZFS integration has been done both with NFS and Samba (CIFS).
Like any other action concerning datasets, properties are sets and unset via the zfs command. On our Funtoo box running zfs-Fuse we can, for example, start by seeing the value of all properties for the dataset myfirstpool/myfirstDS:
# zfs get all myfirstpool/myfirstDS zfs get all myfirstpool/myfirstDS NAME PROPERTY VALUE SOURCE myfirstpool/myfirstDS type filesystem - myfirstpool/myfirstDS creation Sun Sep 4 23:34 2011 - myfirstpool/myfirstDS used 73.8M - myfirstpool/myfirstDS available 5.47G - myfirstpool/myfirstDS referenced 73.8M - myfirstpool/myfirstDS compressratio 1.00x - myfirstpool/myfirstDS mounted yes - myfirstpool/myfirstDS quota none default myfirstpool/myfirstDS reservation none default myfirstpool/myfirstDS recordsize 128K default myfirstpool/myfirstDS mountpoint /myfirstpool/myfirstDS default myfirstpool/myfirstDS sharenfs off default myfirstpool/myfirstDS checksum on default myfirstpool/myfirstDS compression off default myfirstpool/myfirstDS atime on default myfirstpool/myfirstDS devices on default myfirstpool/myfirstDS exec on default myfirstpool/myfirstDS setuid on default myfirstpool/myfirstDS readonly off default myfirstpool/myfirstDS zoned off default myfirstpool/myfirstDS snapdir hidden default myfirstpool/myfirstDS aclmode groupmask default myfirstpool/myfirstDS aclinherit restricted default myfirstpool/myfirstDS canmount on default myfirstpool/myfirstDS xattr on default myfirstpool/myfirstDS copies 1 default myfirstpool/myfirstDS version 4 - myfirstpool/myfirstDS utf8only off - myfirstpool/myfirstDS normalization none - myfirstpool/myfirstDS casesensitivity sensitive - myfirstpool/myfirstDS vscan off default myfirstpool/myfirstDS nbmand off default myfirstpool/myfirstDS sharesmb off default myfirstpool/myfirstDS refquota none default myfirstpool/myfirstDS refreservation none default myfirstpool/myfirstDS primarycache all default myfirstpool/myfirstDS secondarycache all default myfirstpool/myfirstDS usedbysnapshots 18K - myfirstpool/myfirstDS usedbydataset 73.8M - myfirstpool/myfirstDS usedbychildren 0 - myfirstpool/myfirstDS usedbyrefreservation 0 - myfirstpool/myfirstDS logbias latency default myfirstpool/myfirstDS dedup off default myfirstpool/myfirstDS mlslabel off -
How can we set a limit that prevents myfirstpool/myfirstDS to not use more than 1 GB of space in the pool? Simple, just set the quota property:
# zfs set quota=1G myfirstpool/myfirstDS # zfs get quota myfirstpool/myfirstDS NAME PROPERTY VALUE SOURCE myfirstpool/myfirstDS quota 1G local
May be something poked your curiosity: what "SOURCE" means? "SOURCE" describes how the property has been determined for the dataset and can have several values:
- local: the property has been explicitly set for this dataset
- default: a default value has been assigned by the operating system if not explicitely set by the system adminsitrator (e.g SUID allowed or not in the above example).
- dash (-): not modifiable intrinsic property (e.g. dataset creation time, whether the dataset is currently mounted or not, dataset space usage in the pool, average compression ratio...)
Before copying some files in the dataset, let's fix a binary (on/off) property:
# zfs set compression=on myfirstpool/myfirstDS
Now try to put more than 1GB of data in the dataset:
# dd if=/dev/zero of=/myfirstpool/myfirstDS/one-GB-test bs=2G count=1 dd: writing `/myfirstpool/myfirstDS/one-GB-test': Disk quota exceeded
Permission delegation
ZFS brings a feature known as delegated administration. Delegated administration enables ordinary users to handle administrative tasks on a dataset without being administrators. It is however not a sudo replacement as it covers only ZFS related tasks such as sharing/unsharing, disk quota management and so on. Permission delegation shines in flexibility because such delegation can be handled by inheritance though nested datasets. Pewrmission deleguation is handled via zfs through its allow and disallow options.
Data redundancy with ZFS
Nothing is perfect and the storage medium (even in datacenter-class equipment) is prone to failures and fails on a regular basis. Having data redundancy is mandatory to help in preventing single-points of failure (SPoF). Over the past decades, RAID technologies were powerful however their power is precisely their weakness: as operating at the block level, they do not care about what is stored on the data blocks and have no ways to interact with the filesystems stored on them to ensure data integrity is properly handled.
Some statistics
It is not a secret to tell that a general trend in the IT industry is the exponential growth of data quantities. Just thinking about the amount of data Youtube, Google or Facebook generates every day taking the case of the first some statistics gives:
- 24 hours of video is generated every minute in March 2010 (May 2009 - 20h / October 2008 - 15h / May 2008 - 13h)
- More than 2 billions views a day
- More video is produced on Youtube every 60 days than 3 major US broadcasting networks did in the last 60 years
Facebook is also impressive (Facebook own stats):
- over 900 million objects that people interact with (pages, groups, events and community pages)
- Average user creates 90 pieces of content each month (750 millions users active)
- More than 2.5 million websites have integrated with Facebook
What is true with Facebook and Youtube is also true with many other cases (think one minutes about the amount of data stored in iTunes) especially with the growing popularity of cloud computing infrastructures. Despite the progress of the technology a "bottleneck" still exists: the storage reliability is nearly the same over the years. If only one organization in the world generate huge quantities of data it would be the CERN (Conseil Européen pour la Recherche Nucléaire, now officially known as European Organization for Nuclear Research) as their experiments can generate spikes of many terabytes of data within a few seconds. A study done in 2007 quoted by a ZDNet article reveals that:
- Even ECC memory cannot be always be helpful: 3 double-bit errors (uncorrectable) occurred in 3 months on 1300 nodes. Bad news: it should be zero.
- RAID systems cannot protect in all cases: monitoring 492 RAID controller for 4 weeks showed an average error rate of 1 per ~10^14 bits, giving roughly 300 errors for every 2.4 petabytes
- Magnetic storage is still not reliable even on high-end datacenter class drives: 500 errors found over 100 nodes while writing 2 GB file to 3000+ nodes every 2 hours then read it again and again for 5 weeks.
Overall this means: 22 corrupted files (1 in every 1500 files) for a grand total of 33700 files holding 8.7TB of data. And this study is 5 years old....
Source of silent data corruption
Not an exhaustive list but we can quote:
- Cheap controller or buggy driver that does not reports errors/pre-failure conditions to the operating system;
- "bit-leaking": an harddrive consists of many concentric magnetic tracks. When the hard drive magnetic head writes bits on the magnetic surface it generates a very weak magnetic field however sufficient to "leak" on the next track and change some bits. Drives can generally, compensate those situations because they also records some error correction data on the magnetic surface
- magnetic surface defects (weak sectors)
- Hard drives firmware bugs
- Cosmic rays hitting your RAM chips or hard drives cache memory/electronics
Building a mirrored pool
ZFS/RAID-Z vs RAID-5
RAID-5 is very commonly used nowadays because of its simplicity, efficiency and fault-tolerance. Although the technology did its proof over decades, it has a major drawback known as "The RAID-5 write hole". if you are familiar with RAID-5 you already know that is consists of spreading the stripes across all of the disks within the array and interleaving them with a special stripe called the parity. Several schemes of spreading stripes/parity between disks exists in the natures, each one with its own pros and cons, however the "standard" one (also known as left-asynchronous) is:
Disk_0 | Disk_1 | Disk_2 | Disk_3 [D0_S0] | [D0_S1] | [D0_S2] | [D0_P] [D1_S0] | [D1_S1] | [D1_P] | [D1_S2] [D2_S0] | [D2_P] | [D2_S1] | [D2_S2] [D2_P] | [D2_S0] | [D2_S1] | [D2_S2]
The parity is simply computed by XORing the stripes of the same "row", thus giving the general equation:
- [Dn_S0] XOR [Dn_S1] XOR ... XOR [Dn_Sm] XOR [Dn_P] = 0
This equation can be rewritten in several ways:
- [Dn_S0] XOR [Dn_S1] XOR ... XOR [Dn_Sm] = [Dn_P]
- [Dn_S1] XOR [Dn_S2] XOR ... XOR [Dn_Sm] XOR [Dn_P] = [Dn_S0]
- [Dn_S0] XOR [Dn_S2] XOR ... XOR [Dn_Sm] XOR [Dn_P] = [Dn_S1]
- ...and so on!
Because the equations are a combinations of exclusive-or, it is possible to easily compute a parameter if it is missing. Let say we have 3 stripes plus one parity composed of 4 bits each but one of them is missing due to a disk failure:
- D0_S2 = <missing>
However we know that:
- D0_S0 XOR D0_S1 XOR D0_S2 XOR D0_P = 0000 also rewritten as:
Applying boolean algebra it gives: D0_S2 = 1011 XOR 0010 XOR 0110 = 1111. Proof: 1011 XOR 0010 XOR 1111 = 0110 this is the same as D0_P
'So what's the deal?' Okay now the funny part, forgot the above hypothesis and imagine we have this:
Applying boolean algebra magics gives 1011 XOR 0010 XOR 1101 => 0100. Problem: this is different of D0_P (0110). Can you tell which one (or which ONES) of the four terms lies? If you find a mathematically acceptable solution, found your company because you have just solved a big computer science problem. If humans can't solve the question, imagine how hard it is for the poor little RAID-5 controller to determine which stripe is right and which one lies and the resulting "datageddon" (i.e. massive data corruption on the RAID-5 array) when the RAID-5 controller detect error and start to rebuild the array.
This is not science fiction, this a pure reality and the weakness stays in the RAID-5 simplicity. Here is how it can happen: an urban legend with RAID-5 arrays is that they update stripes in an atomic transaction (all of the stripes+parity are written or none of them). Too bad, this is just not true, the data is written on the fly and if for a reason or another the machine where the RAID-5 array has a power outage or crash, the RAID-5 controller will simply have no idea about what he was doing and which stripes are up to date which ones are not up to date. Of course, RAID controllers in servers do have a replaceable on-board battery and most of the time the server they reside in is connected to an auxiliary source like a battery-based UPS or a diesel/gas electricity generator. However, Murphy laws or unpredictable hazards can, sometimes, happens....
Another funny scenario: imagine a machine with a RAID-5 array (on UPS this time) but with non ECC memory. the RAID-5 controller splits the data buffer in stripes, computes a data stripe and starts to write them on the different disks of the array. But...but...but... For some odd reason, only one bit in one of the stripes flips (cosmic rays, RFI...) after the parity calculation. Too bad too sad, one of the written stripes contains corrupted data and it is silently written on the array. Datageddon in sight!
Not to make you freaking: storage units have sophisticated error correction capability (a magnetic surface or an optical recording surface is not perfect and reading/writing error occurs) masking most the cases. However, some established statistics estimates that even with error correction mechanism one bit over 10^16 bits transferred is incorrect. 10^16 is really huge but unfortunately in this beginning of the XXIst century with datacenters brewing massive amounts of data with several hundreds to not say thousands servers this this number starts to give headaches: a big datacenter can face to silent data corruption every 15 minutes (Wikepedia). No typo here, a potential disaster may silently appear 5 times an hour for every single day of the year. Detection techniques exists but traditional RAID-5 arrays in them selves can be a problem. Ironic for a so popular and widely used solution :)
If RAID-5 was an acceptable trade-off in the past decades, it simply made its time. RAID-5 is dead? *Horray!*
More advanced topics
ZFS Intention Log (ZIL)
Final words and lessons learned
ZFS surpasses by far (as of September 2011) every of the well-known filesystems around there: none of them propose such an integration of features and certainly not with this management simplicity and robustness. However in the Linux world it is definitely a no-go in the short term especially for production systems. The two known implementations are not ready for production environments and lacks some important features or behave in a clunky manner, this is absolutely correct as none of them pretend to be at this level of maturity and the licensing incompatibility between the code opened by Sun Microsystems some years ago and the GNU/GPL does not help the cause. However, both look very promising once their corners will become rounded.
For a Linux system, the nearest plan B is you seek for a BTRFS like filesystem covering some of the functionalities offered by ZFS is BTRFS (still considered as experimental, be prepared to a disaster sooner or later although BTRFS is used by some Funtoo core team members since 2 years and proved to be quite stable in practise). BTRFS however does not pushes the limits as much as ZFS does: it does not have built-in snapshot differentiation tool nor implement built-in filesystem streaming capabilities and roll-backing a BTRFS subvolume is a bit more manual than in "the ZFS way of life".
Footnotes & references
Source: solaris-zfs-administration-guide


text:
You have no items in your shopping cart.
You're currently on:
Our compatible Panasonic KX-FA135 Fax Ribbon with Frame and KX-FA136 Refill Rolls (2pk) will meet or exceed OEM (Original Equipment Manufacturer) specifications and high standards of quality and reliability for your Panasonic KX-FM220. Designed to be 100% compatible and interchangeable with OEM cartridges.
2 Item(s) | Show per page |
View as: List Grid | Sort by |
Compatible Panasonic KX-FA135 KXFA135 Thermal Fax Ribbon Cartridge with 1 Roll for KX-FM205
Regular Price: $18.99
Special Price: $12.99
Compatible Panasonic KX-FA136 KXFA136 Thermal Fax Ribbon Refill Rolls (2pk) for KX-FM205 KX-FM210
Regular Price: $18.99
Special Price: $14.99
2 Item(s) | Show per page |
View as: List Grid | Sort by |

subset: commoncrawl


text:
- Text Size:
- ASmall Text
- AMedium Text
- ALarge Text
We're a nation obsessed with thinness (yet ironically obese) and celebrity weight wars are a favorite fixation. When an actress slims down too much, we race to label her anorexic, but God forbid she gains five pounds or we'll take a picture of her bottom and magnify her cellulite on the cover of a magazine.
The following five women know the cover of a tabloid all too well, and the accompanying headline usually reads something like, "Actress Packs on Pounds" or "Starlet Suddenly Skinny."
We're not talking about celebrities who pack on the pounds or whittle their waistlines for a movie role but about those who (despite fame and fortune) just can't seem to keep their weight in check.
Think you've heard it all when it comes to celebrity weight wars? Read on for our top five celebrity yo-yo dieters ...
No. 5: Jennifer Love Hewitt
OK, so this lovely actress was never really that big, but her weight wars have certainly been public nonetheless.
In her youth, Love was the celebrity that every girl wanted to be and every guy wanted to be with. She managed to keep a slim (albeit curvy) figure over the years, but more recent less-than-flattering tabloid photos of Love in her swimsuit revealed that this "Ghost Whisperer" had become more than pleasantly plump.
In response, Love claimed to be proud of her curves, but (not surprisingly) the unflattering photos were followed a few months later by a cover shot in which she had nixed the flab (with a headline that promised to divulge how she'd lost 20 pounds).
While she's not the first celeb to fluctuate in weight, with that naturally curvy frame, we wouldn't be surprised if Love continues to struggle with the scale.
No. 4: Janet Jackson
Her late brother may have been the king of pop, but Janet Jackson is the undisputed queen of yo-yo dieting.
Despite a successful career, this pop star/actress has had her ups and downs (on the scale that is). One minute she's sporting rock-hard abs on tour and the next day her frumpalicious frame is consuming the cover of a tabloid. A plus-size Janet even appeared on the big screen in the 1996 comedy "The Nutty Professor."
According to interviews, Jackson has struggled with her weight since age 10, when television producers asked her to drop pounds for a role. More recently, Janet packed on the pounds for a movie role (supposedly), but the project fell through.
Amazingly, Janet always manages to drop the weight (reportedly thanks to a strict diet of lean protein and veggies), but she doesn't seem to know how to keep it off.
No. 3: Carnie Wilson
Carnie Wilson has made a career out of being a fat celebrity. Since her initial fame and fortune as part of the 1980s rock trio Wilson Phillips, Carnie has been best known for her struggles with the scale.
In 1999, Carnie topped the scale at more than 300 pounds before undergoing a very public gastric bypass surgery. An initial success (she lost more than 150 pounds), Carnie's weight has fluctuated ever since, exacerbated by two pregnancies. In 2006, she joined the cast of "Celebrity Fit Club" and shed some pounds, but the diet didn't stick and she's now eating her way back to her presurgery weight.
Carnie might be ridiculed for her weight issues, but without them, she might not have a career at all. Her most recent jobs include hosting "The Newlywed Game" on GSN and starring in her own reality series called "Carnie Wilson Unstapled."
No. 2: Kirstie Alley
Best known for her role on the TV show "Cheers," in her heyday Kirstie Alley also had one hot little figure.
Don't believe us? Check out "A Bunny's Tale" (1984), in which she goes undercover as a Playboy bunny. Kirstie's character may have been just trying to get the scoop, but with that body, we wouldn't question whether she belonged in Hugh's mansion.
Since the 1980s, Kirstie's weight has fluctuated more times than we can count. Recently she became a spokeswoman for Jennie Craig and reportedly lost 70 pounds on the diet program. However, her new TV series, "Kirstie Alley's Big Life" (emphasis on BIG), clearly showed that she wasn't able to keep it off.
This isn't the first time that Kirstie's ballooning weight has been the butt of the joke. In 2005 (before the diet contract) she starred in a satirical "reality" show called "Fat Actress."
No. 1: Oprah Winfrey
She may be one of the wealthiest celebrities and the most successful woman in daytime television, but Oprah's many accomplishments and charitable acts are often overshadowed by her ever-fluctuating waistline.
Over the years, Oprah has ballooned from average to obese, and then whittled her way down to slim and trim, only to gain it all back again.
This former daytime diva has been very open about her struggles with food and weight, frequently talking about her experiences on her show and in her magazine. No doubt she's helped hundreds or even thousands of Americans trim down, thanks to the diet secrets she's shared, but on a personal level she just can't seem to win the battle.
Whether you feel pity or disdain for these celebrity yo-yo dieters, we're quite certain this won't be the last you hear about their weight wars.
Distributed by Internet Broadcasting. This material may not be published, broadcast, rewritten or redistributed.


text:
Welcome to Smogon! Check out the Smogon Info Hub for everything you need to know about starting out in the community. Don't forget to introduce yourself in the Introduction Thread, too!
Welcome to Smogon Forums! Please take a minute to read the rules.
Rating Basics
Discussion in 'Rating Activities' started by Jimbon, Dec 15, 2011.
Page 13 of 20
Page 13 of 20


text:
Research before you buy new Land Rover with our new Land Rover comparisons, auto reviews, vehicle rebates and incentives from local Land Rover dealers. Read new Land Rover reviews and get the best deal on a new Land Rover near Wabaunsee County, KS. Showcasing a wide range of Land Rover models from various car manufacturers.
Find the best prices on new Land Rover, nearly new and used cars. Find a used Land Rover for sale near Wabaunsee County, KS. CarClub provides you with a wide range of used cars near you. A large selection of used Land Rovers for sale near Wabaunsee County, KS at the cheapest prices.
Flexible auto finance plans for a new Land Rover purchase. We help you get a low interest rate Land Rover loan near Wabaunsee County, KS. Simple. Fast. Free. Get a free rate quote for your auto financing based on your credit, not on your negotiating skills.
Find out how much you can save on your new Land Rover's Land Rover insurance near Wabaunsee County, KS. Recieve personalized competitive auto insurance quotes from mulitiple agents.


text:
Is there an urgent requirement for information about Adelia Andre? If there is you have come to the best place because USA People Search focuses on offering details such as address, phone number, and email for individuals like Adelia Andre. Additionally, with our large repository of data for anyone with the last name Andre, you will definitely come across the person you are searching for.
To shorten the entire hunting process for finding that specific Adelia, we have arranged all our data into four groups – name/aliases, age, location and possible relatives. This will let you examine all the listings of people with the last name Andre and choose the correct match. If you find a listing that is similar to the Adelia Andre you are looking for, just click on the View Details link to the right of the listing to get all the information we have about the person.
If you face a problem with finding the Adelia you are looking for, you can try searching a second time by using the search box above. A different spelling variation could help, as will entering the person's full name, assuming you know it. Once you find the correct Adelia Andre, click through to the details page to explore the extensive database we provide.


text:
Long Beach, NY, USA < change city >
Vendors & Venues
Local Talk
Real Weddings
Photo Galleries
Scratch Weddings
36 Cooper Square
New York, New York 10003, USA
2 ratings, 2 reviews, 2 weddings
Scratch Weddings is the leading DJ company that has entertained over 6 million guests and performed at over 15,000 events for top brands. Our marquee DJs routinely open for leading recording artists, mix before thousands and perform at events and weddings - just like yours. These are A-List performers who are expert at customizing music to you and your wedding night, mixing a wide variety of genres and keeping all guests, ...
1521 Concord Pike #303
Wilmington, Delaware 19803, United States
Limos/Shuttles, Ceremony & Reception
ShuttleWizard.com provides multiple quotes for airport transportation services ranging from airport shuttle service to private car service. We have partnered with hundreds of providers nationwide and pre-negotiated rates to provide our customers with the most value and highest services. We also provide private transportation services for special events and weddings. Just try us out, call one of our customer satisfaction members,...
Chic Bahamas Weddings
Coordinators/Planners, Ceremony & Reception
Chic Bahamas Weddings is a full-service island award winning wedding planning and design company. We create beautiful tropical island elegant weddings in the Bahamas. Together we'll create your dream Bahamas wedding at boutique hotel venues such as Exuma's Turquoise Cay, among other Bahamas wedding locations. Imagine a beautiful beach or tropical Bahamas weddings ceremony on white sand against a backdrop of crystal blue water...
Wce Carriagehouse
187 Brinckerhoff Court
Englewood, NJ 07631, USA
(201) 568-7701
Ceremony Sites, Reception Sites, Ceremony ...
37 ratings, 39 reviews, 47 weddings
The WCE Carriagehouse is a charming historic carriagehouse available for small weddings in Bergen County New Jersey close to Ne...
Nikos Koutras Entertainm...
Astoria, NY 11103, USA
Bands/Live Entertainment, DJs
1 rating, 1 review, 3 weddings
Nikos Koutras Entertainment is a Greek Wedding band that adds A NEW DIMENSION of GREEK ENTERTAINMENT to the industry of Greek W...


text:
The Senior Legal Adviser:
Dealing with the Permanent Mission of Switzerland on host country matters
The Legal Liaison Office (LLO):
Answering legal questions from UNOG's Division of Administration, various departments and services, and Geneva offices of United Nations funds and programmes Coordinating with the Legal Advisers of the Geneva-based international organizations
On an ad hoc basis, the Senior Legal Adviser coordinates with the Legal Advisers of Geneva-based international organizations on matters of common interest under the respective headquarters agreements.
Participating in various panels and commissions
As an ex officio member in various panels and commissions, the Senior Legal Adviser participates in the sessions of:
LLO works in close cooperation with the Office of Legal Affairs (OLA) at Headquarters on legal matters requiring specialized advice and/or a position from the Legal Counsel of the United Nations or the Secretary-General's approval. These include, inter alia:
ILS is convened on an annual basis within the framework of the International Law Commission. As Director of the International Law Seminar (ILS), the Senior Legal Adviser is responsible for fund-raising, selecting participants, organizing the three-week programme and daily follow-up of the activities of participants. This also includes budget management of the ILS Trust Fund and related certifying officer responsibilities. This Trust Fund should provide 17/18 fellowships to the participants in the Seminar through voluntary contributions from Governments.
Providing administrative support to the International Law Commission
LLO manages the budget of the International Law Commission and provides day-to-day logistical assistance to the members and the secretariat of the ILC, with regard to travel, daily subsistence allowance, insurance matters, etc.


text:
With contrasting sweater-style sleeves, this knitted tunic is the perfect trans-seasonal piece. Wear over jeans for now and with bare legs and flats later.
Search by street, town or postcode:
Please call the store before you visit to check that the item you want is still in stock to avoid disappointment. The stock indicator is a guide and items can go out of stock quickly
Sorry, but we do not have stock available at the moment in your selected store. Please call customer services for more information on 0844 557 3285 if calling from within the UK, or +44 1792 784 150 if calling from overseas.
* Please note the stock checker is only a guide to store stock levels.
* style code - 78BCK
Click and Collect: Collect from a UK store free of charge. Excluding Homeware. UK Standard Delivery: Delivered in 5 – 7 working days - £3.95 (no matter how many parcels we send.) Homeware exclusions apply. UK Express Delivery: Order by noon for next working day delivery - £7 per order. Homeware exclusions apply.
Click here for International Delivery rates + more info
Click here for information on Homeware Delivery and Returns
Click here for information on Returns.
Click here for right to cancel.
Allowed payment methods
Contact us Tweet @FC_HELP for customer queries or call customer services on 0333 400 3285 if calling from within the UK, and +44 1792 784 150 if calling from overseas.


text:
Zaneen Lighting Afef 2 Light Wall Sconce
Special Offer:
24 Issues of Better Homes and Gardens
Included with purchase ($10 value).
Zaneen Lighting Afef 2 Light Wall Sconce
Additional Documents
Weights & Dimensions
Product Type: Flush Mount | Style: Modern |
Finish: 1: Black | Finish: 2: White |
Shade Included: Yes | Bulb Type: Halogen |
Bulb Wattage: 75 Watts (W) | Number of Lights: 2 |
Country of Manufacture: Italy |
ADA Compliant: Yes |
About the Manufacturer
From the onset of Zaneen Lighting in 1981 their goal has been to provide clients with high quality lighting from European crafters. Today, Zaneen continues to introduce unique new models in a variety of styles and price ranges all the while maintaining their original commitment to quality. As a trend setter in the industry, Zaneen Lighting proudly continues to be honored with industry awards attesting to the innovative style and design excellence of their lighting products including wall sconces, pendants, chandeliers, kitchen island lighting, table lamps, floor lamps, flush mounts, semi-flush mounts, and foyer lighting!Decorative includes two distinct philosophies. "Euro Trend" offers traditional handcrafted pieces ranging from cast iron multi-tiered chandeliers embellished with shades formed of natural alabaster, to transitional wrought iron models garnered with stylish Italian glass. A vast array of tastes will be impressed by the availability of coordinating pieces in a selection of fashionable finishes.
In contrast, Decorative also includes the "Design" collections, with an unparalleled unique assortment of trend setting contemporary pieces categorized by an architectural flare and functional design. High tech designs with leading edge engineering make Zaneen's Design collection the industry's innovation standard.
Lighting Systems encompass two complementing product categories. The stylish, elegant, and tasteful designs of "Collections" include contemporary chandeliers with glass and shade options that provide unique personality to any fixture and room. Zaneen's collection fixtures carry a presence that will not go unnoticed.
The Lighting Systems category, refered to as "Systems", provides unmatched flexibility to achieve specific lighting goals, whether they are general illumination, task, accent, or ambient lighting. With a range of different line voltage and low voltage systems, and an incredibly diverse and growing selection of light sources and glass, no other North American source comes close to offering the versatility that Zaneen provides with Systems.
"Outdoor" is Zaneen's line of specification-grade outdoor lighting. These modular components and accessories can be assembled for most applications including in-grade, pathway, floodlight, drive-over, and underwater. With optional IP68 compliance, Zaneen's enclosures meet the highest level of environmental protection available and promise no internal condensation, assuring a cleaner, brighter light source over the long term. Zaneen's fixtures also include internal temperature technology allowing a wide variety of possible lamp configurations, and safe operation in fountain or tidal applications where water levels change between wet and dry conditions. Available double glass options ensure that publicly accessible fixtures remain safe to the touch. Zaneen Lighting "Outdoor" has raised the benchmark for industry standards!
Zaneen Lighting is able to address the needs of their clients with highly trained customer service staff, manufacturer's representative agencies throughout North America, and facilities in excess of 100,000 square feet including home base in Toronto, satellite warehouse in Buffalo, and showrooms in Dallas and Toronto.
Zaneen Lighting takes great pride in their past, are evolving in the present, and look forward to the future. Nothing is taken for granted in their commitment to exceptional quality, value, and professionalism.
Zaneen Lighting Afef 2 Light Wall Sconce
Still have questions about this product? We're here to help
Tell Us What You Think!
Tell us what you think about this product. It will help us provide you with better products in the future and help your fellow shoppers make better purchases.
Write the first review Reference
Afef 2 Light Wall Sconce is also sometimes listed under the following manufacturer numbers: D8-3125, D8-3126


text:
NKSJ Holdings, Inc. - SWOT, Strategy and Corporate Finance Report, is a source of comprehensive company data and information. The report covers the company's structure, operation, SWOT analysis, product and service offerings, detailed financials, and corporate actions, providing a 360deg view of the company. This Report Features: * Detailed information on NKSJ Holdings, Inc. required for business and competitor intelligence needs * A study of the major internal and external factors affecting NKSJ Holdings, Inc. in the form of a SWOT analysis * An in-depth view of the business model of NKSJ Holdings, Inc. including a breakdown and examination of key business segments * Five-year financial information and charts, including balance sheet, income statement, cash flows, market capitalization and multiples, and
Author:Bill Thompsone-mailWeb: www.fastmr.comPhone: 18008448156


text:
There are no items in your cart at this time.
Buy 3, Get 1 FREE: SAVE on All Feetures & Thorlo Socks! — Shop Now —
Select your desired product options and we will notify you when it is back in stock.
Johnston & Murphy is a pillar of classic workmanship and a beacon of American style. Since 1850, J&M has provided the highest quality products and set the standard for successful, professional men everywhere.
The Johnston & Murphy Colvard Moc Toe is a rugged waterproof casual that keeps you dry and comfortable no matter what Mother Nature dishes out. Waterproof leather and seam-sealed construction lock out moisture. XC4® construction offers advanced cushioning and a customizable fit.
Johnston & Murphy Colvard Moc Toe Mahogany Leather Men's Shoe
Features include:


text:
Compare Products: Check up to four products with to see a side-by-side comparison
Eco-savvy patrons will love our biodegradable bamboo plates and selection of wood flatware! Our bamboo flatware and dinnerware is durable, biodegradable, and Certified Organic. Each piece is crafted and designed to showcase the bamboo's natural beauty without sacrificing any of the traditional functionalities. Our disposable wooden flatware is made from replanted/harvested birch forests that are F.S.C. (Forest Stewardship Council) Certified, and we offer products made from renewable fallen palm leaves that are biodegradable, compostable and 100% natural. For more eco-friendly solutions, be sure to check out our recycled paper hot cups, biodegradable plastic flatware and other green restaurant supplies!
Other Eco-Friendly Disposables


text:
For the second episode of Land of Lola, Kinky Boots star Billy Porter invites fans inside the recording studio for a sneak peek at the cast album. Also, get ready for Sunday service at "the church of Kinky."
Laura Michelle Kelly has her camera hijacked by Matthew Morrison in a very special behind-the-scenes video at Finding Neverland.
April 18, 2015Roahl Dahl's James and the Giant Peach is now a musical by Tony nominated songwriters Benj Pasek and Justin Paul! Check out our exclusive video from the recording studio and download your own FREE copy of the album here!
April 21, 2015Hate Shakespeare? Love musicals? You're in luck! Check out this footage of Something Rotten!, starring Brian d'Arcy James and Christian Borle.
April 22, 2015It's opening night at It Shoulda Been You and Sierra Boggess takes us there...until her camera is stolen by Neil Patrick Harris.
April 21, 2015Check out video clips of the new Broadway musical Doctor Zhivago, starring Tam Mutu, Kelli Barrett, Paul Alexander Nolan and Tom Hewitt.
April 21, 2015Disney has done it again with their new stage adaptation of the animation hit Aladdin! Hear all about the new family favorite from the stars and creators in this video trailer.
July 15, 2014Disney's smash hit adapation of the beloved animated film is a Broadway sensation. Find out why!
November 10, 2014Learn all about the Tony-winning smash hit musical comedy that's one of Broadway's biggest blockbusters ever.
October 24, 2014Broadway.com is having a blast on Periscope, the new app that allows you to stream live. On opening night, we asked An American in Paris star Brandon Uranowitz to document his day and here are the highlights!
April 16, 2015Fun Home features three performers (Beth Malone, Emily Skeggs & Sydney Lucas) playing the central character in this poignant musical memoir.
April 20, 2015Find out why audiences have fallen in love with this fantastical musical inspired by The Wizard of Oz.
October 19, 2014It's a long day at Finding Neverland as Laura Michelle Kelly and the cast get up super early to be on Good Morning America.
April 25, 2015


text:
Send an article to a friend by filling out the form below. Fields marked with *are required.
Article Title: New Members - April 2007
Month of Issue: April 2007
Article Description:
* Your Name:
* Send this Article to:
Add your own personal message:
*Enter your email address:
What would you like to send?
Please verify that you are human:


text:
The original deal pioneer keeps on pinching pennies and cracking wise. No, that price isn't a typo.
Go to Woot
Candlepower and puppet shows are so passé. Get the newest TVs, speakers, home theater gear, and more.
Go to Electronics.Woot
The robots are coming! And so are the laptops, the tablets, the chargers, and all sorts of other gadgets!
Go to Computers.Woot
High living at low prices with name-brand deals for your pad, crib, nest, or castle - inside and out.
Go to Home.Woot
Make it, build it, improve it. But whatever home project you've got in mind, start by saving money on it.
Go to Tools & Garden
Sports, fitness, and outdoors deals, because saving money is only one of your passions.
Go to Sport.Woot
You don't have to spend a lot to look like you do. Werk it and twerk it with amazing deals on designer watches, handbags and more.
Go to Accessories & Watches
Deals for moms and dads and anybody else who needs kids' stuff, from toys to baby gear to furniture to clothing.
Go to Kids.Woot
Our exclusive graphic tees will make you laugh, will make you cry, will make you look totally hot.
Go to Shirt.Woot
Direct from the winery to your thirsty lips: wine deals worth doing a spit-take over.
Go to Wine.Woot
Last chance deals for hardcore cheapskates. Catch 'em before they - you guessed it - sell out.
Go to Sellout.Woot
460,641 deals (and counting) from around the web, shared and ranked by a community of deal fiends like you. Go to Deals.Woot.
ThunderThighs wrote:amplitude modulation
volume control
pooflady wrote:volume control
mute button
AZGman wrote:mute button
moot point
no1 wrote:moot point
useless query
pooflady wrote:useless query
wasted breath
AZGman wrote:wasted breath
political argument
pooflady wrote:political argument
rhetorical bombast
AZGman wrote:rhetorical bombast
much ado
pooflady wrote:much ado
about nothing
ThunderThighs wrote:about nothing
black hole
pooflady wrote:black hole
emotional need
AZGman wrote:emotional need
codependent relationships
ThunderThighs wrote:codependent relationships
sponge people
AZGman wrote:sponge people
freaky skin
ThunderThighs wrote:freaky skin
flakey skin
AZGman wrote:flakey skin
dandruff shampoo
ThunderThighs wrote:dandruff shampoo
real poo
AZGman wrote:real poo
rubber dookey
ThunderThighs wrote:rubber dookey
whoopee cushion
AZGman wrote:whoopee cushion
fake farts
pooflady wrote:fake farts
baked tarts
no1 wrote:baked tarts
tiny pies
AZGman wrote:tiny pies
dollhouse food
ThunderThighs wrote:dollhouse food
mud pies
pooflady wrote:mud pies
dirt clods
AZGman wrote:dirt clods
dumb rednecks
no1 wrote:dumb rednecks
white trash
pooflady wrote:white trash
paper waste
AZGman wrote:paper waste
sewage plant
andocommanndo wrote:sewage plant
recycling center
pooflady wrote:recycling center
garbage can
jawsuser wrote:garbage can
raccoon diner
pooflady wrote:raccoon diner
front porch
jawsuser wrote:front porch
rocking chair
andocommanndo wrote:rocking chair
cat's tail
AZGman wrote:cat's tail
pond plant
ThunderThighs wrote:pond plant
peat bog
AZGman wrote:peat bog
potting soil
ThunderThighs wrote:potting soil
clean dirt
AZGman wrote:clean dirt
mud wrestling


text:
Reload the page and try again.
Online Courses: This is not exactly online courses, but they do have you use an ATI testing website for class assignments which is very helpful for learning and preparing for the NCLEX
{{review.SectionLabel}} at {{review.EntityName}}:
Sorry, there are no reviews.
No summaries have been written about this school yet. If you are a student at this school, you should take the college student survey.
Non-registered users are limited to 5 school profile page views per month.
Register for free to gain full access!


text:
Job Zone Information for:
47-3013.00 - Helpers--Electricians
Bright Outlook
Job Zone Two: Some Preparation Needed
Education | These occupations usually require a high school diploma. |
Related Experience | Some previous work-related skill, knowledge, or experience is usually needed. For example, a teller would benefit from experience working directly with the public. |
Job Training | Employees in these occupations need anywhere from a few months to one year of working with experienced employees. A recognized apprenticeship program may be associated with these occupations. |
Job Zone Examples | These occupations often involve using your knowledge and skills to help others. Examples include sheet metal workers, forest fire fighters, customer service representatives, physical therapist aides, salespersons (retail), and tellers. |
SVP Range | (4.0 to < 6.0) |


text:
Currently Shopping For Parts and Gear for Your:
1986 Honda XR250R
BikeBandit.com offers the widest range of Throttles from all of the top leading brands at our best price guarantee.New aftermarket Throttles can help to improve the performance of your motorcycle or ATV, keeping your ride running in peak condition. BikeBandit.com offers a complete selection of Throttles at our best price guarantee. Finding the Throttles for your machine is easy at BikeBandit.com. Simply use the navigation below to select your machines make, year and model.BikeBandit.com is the Web's Largest Powersports Store with more than 8 million factory fresh motorcycle parts and accessories online, so of course we have the most up to date Throttles for you to choose from. Whether you're a recreational biker on the weekend, a serious motocross or dirt bike racer, or a seasonal ATV enthusiast we have the Throttles to help you get riding.We know when you order your Throttles you'll want them now, so BikeBandit.com has developed the best online real-time order tracking system in the powersports industry. You can keep track of every step of your order at BikeBandit.com and once it has been shipped, we provide package tracking information from UPS, FedEx or USPS so you know exactly when your Throttles will arrive.For the Guaranteed Best Price on Throttles, with the highest possible level of customer service, choose BikeBandit.com.


text:
Dover Corporation - SWOT, Strategy and Corporate Finance Report
- Language: English
- Published: January 2013
A textual analysis of the financial results for Dover Motorsports, Inc. compared to selected competitors. Also included are quantitative analyses of the company's financial statements, extensive ratio analyses and up to 10-year history of sales, earnings, dividends and security pricing. A business description of the company, including contact information, senior officers and Wright Quality Rating analyses is also provided.
Quantitative analyses of financial statements (Income Statement, Balance Sheet and Sources of Capital), extensive ratio tables (Accounting, Asset Utilization, Employee Efficiency, Fixed Charges Coverage, Leverage Analysis, Liquidity Analysis, Per-share Data and Profitability Analysis) and proprietary Wright Quality Rating analyses tables. The Wright Quality Rating, a measure of corporate risk, examines the liquidity of the stock, financial strength, profitability and corporate growth characteristics of Dover Motorsports, Inc. There is a textual review of the financial results of Dover Motorsports, Inc. versus Canterbury Park Holding Corporation, Lakes Entertainment, Inc. and Nevada Gold & Casinos, Inc. The comparative analysis of these companies compares
Company Fundamentals
- Company Profile
- Comparative Business Analysis
- Summary Analysis per Share
- Sales Analysis
- Price Analysis
- Earnings and Dividends Analysis
Financial Statement Analyses
- Balance Sheet - Common Size
- Balance Sheet - Year-Year Percent Change
- Balance Sheet - Five-Year Averages
- Income Statement - Common Size
- Income Statement - Year-Year Percent Change
- Income Statement - Five-Year Averages
- Sources of Capital
Financial Ratio Analyses
- Accounting Ratios
- Asset Utilization
- Employee Efficiency
- Fixed Charges Coverage
- Leverage Analysis
- Liquidity Analysis
- Per-Share Data
- Profitability Analysis
Wright Quality Rating Analyses
- Investment Acceptance
- Financial Strength
- Profitability and Stability
- Corporate Growth
United States


text:
Steinbach A Christmas Carol Charles Dickens Nutcracker. 200th Anniversary.
- Product Description
The Steinbach A Christmas Carol Charles Dickens Nutcracker 200th Anniversary is even more elegant than the first. A Christmas Carol's 200th anniversary has come and gone and what better way to celebrate it than with this beautifully handcrafted nutcracker. He looks almost real with his quill and books and is one of only 5,000 created. This piece is approximately 18" tall. Introduced in 2012.
- Product Specs
Brand: Steinbach Collection: A Christmas Carol Year: 2012 Material: Wood Height: 11 inches Feature: Steinbach 200th Anniversary Charles Dickens Christmas Carol Nutcracker Feature: Approximately 18 inches tall Feature: 200th Anniversary Piece Feature: Limited Edition of 5,000 pieces Feature: Handmade, carved and painted in Germany - Shipping and Returns
Free Shipping Available
Normally ships in 3 days. Expected ship date: 5/12/2015.
Get Delivery Estimates Shipping Info:
Standard shipping in the continental US is $7. This method will have your items delivered within 4 business days 98% of the time. Orders placed Monday - Friday before 4 pm (CST) will go out the same day, all other times orders will be sent out the next business day.
If you need your shipment to have a guaranteed delivery, then select one of our other services, 3-day, 2nd day or next day. These services vary in price based on the size and weight of your shipment and the final destination of the package. To find out the exact cost of shipping for your order, please add the items you want to your cart, go to the shopping cart screen and enter in your zip code in the appropriate field. You will be given a shipping estimate. For more information on shipping, please visit our shipping information pages.
Our Guarantee:
We strive to deliver delight to every customer. If for any reason you are not 100% delighted with your purchase, please contact us. You may return your item(s) within 60 days of the original purchase date. You will be refunded the cost of the item(s) in the original form of payment. Additional information about our return process can be found here.
Average Customer Review: 0 out of 5 Stars (0 Reviews)
Brand Details
Steinbach Nutcrackers : A Family Tradition According to the late Herr Christian Steinbach, patriarch of the Steinbach nutcracker family, "A nutcracker is a symbolic figure, one that gives happiness and protects your home and family." The Steinbach family knows this well and has been producing fine wood products in Germany since 1832. We feature a large selection of nutcrackers signed by Christian Steinbach (fifth generation), Karla Steinbach (sixth generation) and Karolin Steinbach (seventh generation). We are independent Steinbach dealer, proud to stock the world's largest selections of German Steinbach nutcrackers. We're stocking the entire 2013 Steinbach nutcracker collection. Our Steinbach store features nutcrackers from many different Steinbach collections, including Wizard of Oz, A Christmas Carol, Disney, 12 Days of Christmas, and many more. We also have a large selection of Chubby, Troll and Mini Steinbach nutcrackers, and ornaments. The name Steinbach is recognized the world over for their unique wood carved and hand painted creations developed over generations by the Steinbach Family. Steinbach Nutcrackers are the most popular German nutcrackers in the world and are handed down from generation to generation. The nutcrackers vary greatly in price depending on their collectibility and availability. |
More from this Brand
Steinbach 17" Paul Bunyan Limited Edition Signed Nutcracker
Special Price $149.97
Was: $449.95 (67% savings)
More Collections from this Brand
- Collection 12 Days of Christmas
- Collection A Christmas Carol
- Collection Christmas Legends
- Collection Christmas Traditions
- Collection Chubby
- Collection Chubby Nutcracker Suite
- Collection Chubby Occupational
- Collection Chubby Oktoberfest
- Collection Chubby Storybook Characters
- Collection Chubby Wizard of Oz
- Collection Collector's Club
- Collection German King's Court