Comments on: TCP Trace Analysis for NFS

By: Kyle Hailey

Kyle Hailey — Tue, 16 Jul 2013 18:13:15 +0000

Hi Mimmus, check out a previous blog post http://dboptimizer.com/2011/07/19/nfs-throughput-testing-trick-solaris/ on using dd with ramdisk and/or memory mapped files to cut out the file system. For dd and caching, on Linux you can use flags like

write
dd if=/dev/zero of=toto bs=8k count=100 oflag=direct
read
dd of=/dev/null if=toto bs=8k count=100 iflag=direct

By: mimmus

mimmus — Tue, 16 Jul 2013 07:17:28 +0000

I understand.
I think that cache on the NFS server (an EMC filer) distorts these simplistic tests.

By: Kyle Hailey

Kyle Hailey — Thu, 11 Jul 2013 23:11:53 +0000

@mimmus:
Have you tried using netio, netsurf or iperf instead of dd?
Have you tried bs=32768?
A bs=1024 is small.
if “mmm” is a location on an NFS server and
NFS transfers are going to take around 200us on the fast side.
and if a transfer is 1024 that is 1K then that’s 5k/ms or 5M/sec 300MB/min which is even slower than you are seeing.
In this case I would guess there is some aggregation going on already to hit 1GB/m
If your NFS exchanges are 12ms then the transfer sizes have to be around 200K
So in that case why not increase your bs explicitly?

For transfer rates it depends on the layers :
program, dd in this case
os, nfs client
tcp nfs client
nic, nfs client
network
nic nfs server
tcp nfs server
os nfs server
nfs on nfs server
I/O nfs server
you want to see which layer the time is being spent

you are doing a dd write from nfs client to nfs server so first big question is how fast is the I/O write speed on the NFS server.
The script on this blog post isolates latency at the nfs client tcp layer to the nfs server tcp layer
if both those layers are slow then the problem is on the nfs server at either the nfs server os, nfs code path or I/O
The script on this blog post doesn’t look at transfer sizes. It should for cases like this. In my case I knew the transfer sizes were 8K. In your case the transfer size has to be bigger than 1K if you are seeing 1GB/min with 12-13ms NFS exchanges.

By: mimmus

mimmus — Wed, 10 Jul 2013 08:37:40 +0000

Hi, the test command is:
dd if=/dev/zero of=mmm bs=1024 count=1000000

The transfer occurs on a 10 Gbps network but it lasts +60 secs!!! On another server with same setup, it lasts 4-5 secs!
Mount options are:
rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,vers=3
(Oracle suggests these).

I tested network using iperf and I can achieve almost 10 Gbps full, the bottleneck is elsewhere.

By: Kyle Hailey

Kyle Hailey — Sun, 07 Jul 2013 23:38:11 +0000

@mimmus: what exactly is taking 12-13ms? Is that the total transfer time at the TCP level or is that the transfer time of NFS request? and if it’s the tranfer time of one NFS request then the question is, what is the size of the NFS request.
what is the syntax of the dd command you are running? The important is the block size, ie “bs=” part. Also are you doing a dd that is writing to the NFS server or reading from the NFS server.
A dd 1 GB of a 1 GbE should be around 100 MB/s thus around 10 secs, but if I change the bs size to something small, I can end up slowing the transfer time. The NFS I/O request size can be up to 1MB if the mount options are set the high and the OS supports it (HP and LINUX support 1MB, Solaris does too but you have to change a kernel parameter and AIX only supports 64K unless you use a patch to increase it)
If you just want to see what the throughput is across the wire then it might be better to use a tool like netio or netsurf which will test the throughput at different I/O transfer sizes over TCP.
In the above example I know the NFS request size is one database datablock size which is 8K

By: mimmus

mimmus — Fri, 05 Jul 2013 14:47:45 +0000

I’m trying to use this script to understand why simply “dd”-ing a 1 GB file lasts more than 60 seconds. If write latency match on server and client and it’s about 12-13 ms, where can be the problem???