6.824 - Spring 2012

6.824 Lab 2: Basic File Server

Due: Friday, February 24th, 5:00pm.

Introduction

In this lab, you will start your file system implementation by getting the following FUSE operations to work: Recall that YFS has the following architecture.

We provide you with skeleton code for both the YFS and extent server modules above.

The YFS module implements the core file system logic. This module runs as a single process called yfs_client that supports a mountpoint on the local host. The code skeleton for this module consists of two pieces:

The extent server stores all the data representing your filesystem, much like a hard disk in an ordinary file system. In later labs, you will run YFS clients on multiple hosts, all of them talking to the same extent server; the desired effect is that all the hosts see and share the same file system. The only way the multiple YFS clients can share data is through the extent server. The extent server code skeleton consists of two pieces:

Getting started

First, use a computer with FUSE and the FUSE libraries installed (see the Lab Overview for details).

Merge your solution to Lab 1 with the new code we supply for Lab 2. To do this, first commit your solution to Lab 1:

% cd lab
% git commit -am 'my solution to lab1'
Created commit 254dac5: my solution to lab1
 3 files changed, 31 insertions(+), 6 deletions(-)
Before you commit, you will need to explicitly git add any new files you added. Run git status to show all the files that were added or changed.

Next, fetch the latest version of the course repository, then create a local branch called lab2 based on our lab2 branch, origin/lab2:

% git pull
remote: Generating pack...
[...]
% git checkout -b lab2 origin/lab2
Branch lab2 set up to track remote branch refs/remotes/origin/lab2.
Switched to a new branch "lab2"
%

The git checkout -b command shown above actually does two things: it first creates a local branch lab2 that is based on the origin/lab2 branch provided by the course staff, and second, it changes the contents of your lab directory to reflect the files stored on the lab2 branch. Git allows switching between existing branches using git checkout branch-name, though you should commit any outstanding changes on one branch before switching to a different one.

You will now need to merge the changes you made in your lab1 branch into the lab2 branch, as follows:

% git merge lab1
Git may not be able to figure out how to merge your changes with the new lab assignment (e.g., if you modified some of the code that the second lab assignment changes). In that case, git merge will tell you which files have conflicts, and you should first resolve the conflicts (by editing the relevant files) and then run git commit -a.

Here's how to run the extent server and yfs_client(s). If you are using the class machines, choose a port number that other students aren't using. For example, to run the extent server on port 3772, type this:

% cd ~/lab
% make
% ./extent_server 3772 &

Next, start the yfs_client process using three parameters: a unique mountpoint (that other students aren't using), the extent server's port number, and the lock server's port number (not used in this lab). The mountpoint must be an empty directory that already exists. To start the yfs_client using mountpoint ./myfs and extent_server that listens on port 3772, type this:

% cd ~/lab
% mkdir myfs
% ./yfs_client ./myfs 3772 3762 &

We provide you with the script start.sh to automatically start extent_server and yfs_client, and stop.sh to kill previously started processes. Actually, start.sh starts two yfs_clients with ./yfs1 and ./yfs2 mountpoints respectively. Thus you can type:

% cd ~/lab
% ./start.sh
% ./test-lab-2-a.pl ./yfs1
% ./test-lab-2-b.pl ./yfs1 ./yfs2
% ./stop.sh

Note: If stop.sh reports "Device or resource busy", please keep executing stop.sh until it says "not found in /etc/mtab", such as:

fusermount: entry for /home/your_name/yfs-class/yfs1 not found in /etc/mtab
fusermount: entry for /home/your_name/yfs-class/yfs2 not found in /etc/mtab
...

The skeleton code implements only GETATTR and STATFS, so the file system you just mounted will not be very useful. You will implement more FUSE operations in this lab to make your file system more functional.

If you are using your own Linux machine, make sure your user ID is in the fuse group so that you can mount without being root. See Lab Overview about how to add a user to the fuse group. On FreeBSD, you can run sysctl vfs.usermount=1 as root, and then you should be able to run yfs_client as a normal user. Most Linux machines at MIT are already configured to allow mount without being root, for example:

Lab 2 is known to work on Leopard, Tiger, and Snow Leopard.

Part 1: CREATE/MKNOD, LOOKUP, and READDIR

Part 1: Your Job

Your job in Part 1 is to implement the extent server, then implement the CREATE/MKNOD, LOOKUP and READDIR FUSE operations in YFS. You must store the file system's contents in the extent server, so that in Part 2 and future labs you can share one file system among yfs_clients. It is OK for your extent server to store data only in memory; this means that if you restart it, all the data previously stored will be lost.

On some systems, FUSE uses the MKNOD operation to create files, and on others, it uses CREATE. The two interfaces have slight differences, but in order to spare you the details, we have given you wrappers for both that call the common routine createhelper(). You should implement createhelper().

As before, if your server passes our tester on the official class programming environment, you are done. If you have questions about whether you have to implement specific pieces of file system functionality, then you should be guided by the tester: if you can pass the tests without implementing something, then you do not have to implement it. For example, you don't need to implement the exclusive create semantics of the CREATE/MKNOD operation. You may modify or add any files you like, except that you should not modify the the tester script or the RPC library.

The Lab 2 tester for Part 1 is test-lab-2-a.pl. Run it with your YFS mountpoint as the argument. Here's what a successful run of test-lab-2-a.pl looks like:

% ./test-lab-2-a.pl ./yfs1
create file-yyuvjztagkprvmxjnzrbczmvmfhtyxhwloulhggy-18674-0
create file-hcmaxnljdgbpirprwtuxobeforippbndpjtcxywf-18674-1
...
Passed all tests!
The tester creates lots of files with names like file-XXX-YYY-Z and checks that they appear in directory listings.

If you implemented at-most-once RPC correctly, the tests should pass with RPC_LOSSY set to 5 as well.

If test-lab-2-a.pl exits without printing "Passed all tests!", then it thinks something is wrong with your file server. For example, if you run test-lab-2-a.pl on the skeleton code we give you, you'll probably see an error message like this:

test-lab-2-a: cannot create /tmp/b/file-ddscdywqxzozdoabhztxexkvpaazvtmrmmvcoayp-21501-0 : No such file or directory

This error message appears because you have not yet provided code to handle the CREATE/MKNOD operation with FUSE. That code belongs in fuseserver_createhelper in fuse.cc.

Note: testing Part 1 on the command line using commands like touch will not work until you implement the SETATTR operation in Part 2. For now, you should do your testing via the creat/open, lookup, and readdir system calls in a language like Perl, or simply use the provided test script.

Part 1: Detailed Guidance

Part 2: SETATTR, READ, WRITE

Part 2: Your Job

Your job in Part 2 is to implement SETATTR, WRITE, and READ FUSE operations in fuse.cc. Once your server passes test-lab-2-b.pl, you are done. If you have questions about whether you have to implement specific pieces of FUSE functionality, then you should be guided by the tester: if you can pass the tests without implementing something, then don't bother implementing it. Please don't modify the test program or the RPC library. We will use our own versions of these files during grading.

test-lab-2-b.pl tests reading, writing, and appending to files, as well as testing whether files written on one yfs_client instance can be read through a second one. To run the tester, first start two yfs_clients using the start.sh script. It runs two yfs_client processes each mounting the same file system under a different name (yfs1 or yfs2).

% ./start.sh

Now run test-lab-2-b.pl by passing the yfs1 and yfs2 mountpoints. Since the script tests the yfs_clients sequentially, you do not need to worry about locking for this lab.

% ./test-lab-2-b.pl ./yfs1 ./yfs2
Write and read one file: OK
Write and read a second file: OK
Overwrite an existing file: OK
Append to an existing file: OK
Write into the middle of an existing file: OK
Check that one cannot open non-existant file: OK
Check directory listing: OK
Read files via second server: OK
Check directory listing on second server: OK
Passed all tests
% ./stop.sh

If test-lab-2-b.pl exits without printing "Passed all tests!" or hangs indefinitely, then something is wrong with your file server. After you are done with Part 2, you should go back and test with test-lab-2-a.pl again to make sure you did not break anything.

Part 2: Detailed Guidance

Handin procedure

E-mail your code as a gzipped tar file to 6.824-submit@pdos.csail.mit.edu by the deadline at the top of the page. To do this, execute these commands:
% cd ~/lab
% ./stop.sh
% make clean
% rm core*
% rm *log
% cd ..
% tar czvf `whoami`-lab2.tgz lab
or
% cd ~/6.824/lab
% make handin
That should produce a file called [your_user_name]-lab2.tgz in your home directory. Attach that file to an email and send it to 6.824-submit@pdos.csail.mit.edu.
Please post questions or comments on Piazza.
Back to 6.824 home.