6.894 Lab 4: User-level NFS3 server

Due date: October 24. Don't wait until the last minute.


This lab is a short lab designed to make you familiar with the SFS software infrastructure. In the previous assignments you have used SFS's async lib. In this assignment you will be exposed to a functional distributed file system, remote procedure call, and the NFS 3 protocol. We expect that most of you wil do a final project in the area of distributed file systems and use the SFS software for your final project; you don't have to but it will save you a lot of time.

This assignment is simple, in principle: you are supposed to fill in some missing blanks in an user-level NFS3 server that we will hand you. At the end of the assignment you will have an operational, usable, secure, distributed file system. The assignment should take you a couple of hours (we did most of the work for you). Once you have completed the assignment, you should start thinking of what you want to do for your final lab. The final lab is essentially to build a cool system of your own design and write a short paper about it. (If you don't have any specific ideas for a project yourself, we will suggest a number of possible projects.) Details will follow shortly.

Although this assignment is easy, you will be handed an arsenal of sophisticated weapons and it also easy to shoot yourself in the foot (i.e., you may have to reboot your workstation). We (that is David Mazieres, the principle author of SFS) went through great length to make the software robust. Programming errors in the client and the server of the file system shouldn't crash your local operating system.

Fetching and compiling sfsusrv

Log into an Athena Sun workstation. You can tell it's a Sun because the output of uname -a will mention "SunOS" and "sparc", like this:
athena% uname -a
SunOS all-night-tool.mit.edu 5.7 Generic_106541-09 sun4u sparc SUNW,Ultra-60
Fetch sfsusrv.tar with your web browser and place it in your Athena home directory. This tar file contains just the source for a basic file server; it depends on library files which we've pre-compiled for you in /mit/6.894/sfs.

Run these commands:

athena% add -f gnu
athena% add 6.894
athena% tar xvf sfsusrv.tar
athena% cd sfsusrv
athena% make
The make may take five or ten minutes.

You need to create a public/private key pair for sfsusrv to use to prove its identity to clients. Run these commands in your sfsusrv directory:

athena% setenv SFS_RUNINPLACE `pwd`
athena% /mit/6.894/sfs/agent/sfskey gen -P sfs_host_key
Type return when sfskey prompts you for the Key Name.

Running sfsusrv

You will need to run both sfsusrv and a client program. You can't run them them on the same machine. The client program must be run as root, so you will need to run it on the Athena workstation you are sitting in front of. You can run sfsusrv either on another workstation, or on a dialup machine; we'll assume the latter.

In order to avoid conflicts between different 6.894 students running sfsusrv on the same dialup machine, you should choose yourself a random port number between 1024 and 65536. Let's call it my-port.

Open a new window and ssh to athena.dialup.mit.edu. Make sure you're on a Sun. Then:

athena% cd ~/sfsusrv
athena% mkdir /tmp/export-$user
athena% rm sfsusrv_config
athena% echo 'keyfile sfs_host_key' > sfsusrv_config
athena% echo "export /tmp/export-$user" >> sfsusrv_config
athena% setenv SFS_RUNINPLACE `pwd`
athena% setenv SFS_PORT my-port
athena% ./sfsusrv -f sfsusrv_config
If all went well, you should see output like this:
sfsusrv: version 0.5, pid 8493
sfsusrv: serving /sfs/biohazard-cafe.mit.edu:dgn84hqianfvqpvh7a42wgubqz9pdy9g
sfsusrv: No sfssd detected, running in standalone mode.
The /sfs/biohazard-cafe.mit.edu:dgn84... is a self-certifying pathname. It contains enough information for an SFS client to contact your SFS server and verify the server's identity: the biohazard-cafe.mit.edu part names the server, and the bytes after the colon are a hash of the server's public key and server name. You can find out more about self-certifying pathnames and SFS at http://www.fs.net; we will also read a paper about SFS (see schedule).

sfsusrv is now acting as an SFS file server for the directory /tmp/export-$user. Now you need to start an SFS client.

Running the SFS client

You must run the SFS client software as root. There can't be more than one SFS client running on any one machine. You should run the software on the Athena machine you're sitting in front of. Type the following:
athena% cd /tmp
athena% mkdir sfs-$user
athena% cd sfs-$user
athena% mkdir etc
athena% cp ~/sfsusrv/etc/* etc
athena% add 6.894
athena% ln -s /mit/6.894/sfs/nfsmounter .
athena% tellme root
athena% su
(Type the root password, which the tellme root
command told you.)
athena% setenv SFS_RUNINPLACE `pwd`
athena% setenv SFS_PORT my-port
athena% /mit/6.894/sfs/sfscd/sfscd
Now open another window on your workstation. You'll need the self-certifying pathname that sfsusrv printed, so copy it with the mouse. Then type
athena% ls
You should see a list of the files in /tmp/export-$user on the sfsusrv machine.

Reading a file

Now try to read a file on the client machine:
athena% echo hi > junk-$user
athena% cat junk-$user
You can't read the file because sfsusrv doesn't implement the read operation. Your job is to implement the NFS3 read RPC. The file that implements the NFS3 RPCs is client.C. Open client.C and you will see how the other NFS3 RPCs are implemented using standard file system system calls; your job is to create a similar implementation for read, perhaps named client::nfs3_read().

If you do this correctly, you will only have to make changes to client.C, but we encourage you to understand how the client and server work. The parts of relevance are the client code, the RPC files, and sfsusrv. You have your copy of the sfsusrv source; everything else (including libraries that sfsusrv uses) is in subdirectories of /mit/6.894/sfs.

The code that runs on the client machine is stored in the directories sfsmounter, sfscd, and sfsrwcd. sfsmounter is the mounter for /sfs. sfsmounter contains very sophisticated code; you don't really have to understand it. The cool thing about it is that you can kill sfscd or sfsrwcd and restart it without having to reboot your computer.

sfscd makes an encrypted, authenticated connection with the server specified in the self-certifying pathname using the FS protocol. The SFS protocol is specified in svc/sfs_prot.x. The makefiles run this file through an RPC compiler to generate stubs and marshalling code; the RPC code is in the directory arpc. The crypto code is in the directory crypt.

If sfscd succeeds in setting up a connection, the remote server will tell it to handoff the connection to sfsrwcd. sfsrwcd relays NFS3 calls issued by the local kernel to the server on the other side of the secure connection (sfsusrv in this case). It does the same for replies. The RPC file for NFS3 is stored in svc/nfs3_prot.x. SFS extends NFS3 slightly to include an expiration field for file attributes to help with cache consistency; this only affects RPC responses, whose definitions are in svc/nfs3exp_prot.x sfsusrv sets the expiration field (called expire) to zero, which causes the SFS client to act just as it would for standard NFS3.

Here are some hints:

Tracing RPCs

After you've added read to sfsusrv correctly, do the following to help you understand how the SFS client and server communicate with RPC.

Stop your sfsusrv with control-C, then re-start it with these commands:

athena% setenv ASRV_TRACE 10
athena% ./sfsusrv -f sfsusrv_config >& sfsusrv.trace
Make sure you're still running sfscd. Type these commands on the sfscd machine:
athena% rm junk-$user
athena% echo hello > junk-$user
athena% cat junk-$user
athena% cat junk-$user
Stop sfsusrv. Now look at the RPC queries and replies recorded in sfsusrv.trace. Which correspond to the creation of junk-$user? Which to the first cat? Which to the second cat? Explain any differences between the RPCs caused by the two cat commands.

How/What to hand in

You should send us an email containing your client.C and your answers to the questions about the RPC traces. Please don't use e-mail attachments. Use the following command:
% cat trace-explanation.txt client.C | Mail -s '6.894 lab 4' 6.894-submit@pdos.lcs.mit.edu
If you have any problems about submission, please contact jinyang@lcs.mit.edu. The lab is due by the beginning of class on Tuesday, October 24.