O/S overview

Overview

6.828 goals:
- Understand operating systems in detail by designing and implementing a small O/S
- Hands-on experience with building systems ("Applying 6.033")
What problems does an O/S solve?
- that is, why not directly program the bare hardware?
- there are a lot of painful h/w details
- it wouldn't be portable
- you'd have to build up lots of functionality for yourself
- it's unlikely that a computer could run more than one app
What's the O/S solution?
- e.g. OSX, Windows, Linux
- the small view: a h/w management library
- the big view: physical machine -> abstract one w/ better properties
- layer picture:
  - h/w: CPU, mem, disk
  - kernel: [various services]
  - user: applications, e.g. vi and gcc
- we care a lot about the interfaces and internel kernel structure
what services does an O/S kernel typically provide?
- processes
- memory
- file contents
- directories and file names
- interprocess communication
- many others: users, security policies, network, time, terminals
what makes a good kernel service design?
- Abstract the hardware for programmer convenience
- Multiplex the hardware among multiple applications
- Isolate applications to contain bugs
- Allow sharing among applications
What does an O/S abstraction look like?
- Applications only see them via system calls
- Examples, from UNIX / Linux:
- fd = open("/dev/foo", 1);
- write(fd, "hello\n", 6);
- pid = fork();
Why is O/S design hard/interesting?
- fast vs abstract
- many features vs few mechanisms
- interactions: fd = open(); ...; fork();
- interactions: CPU priority vs memory allocator.
- open problems: security, multi-core
You'll be glad you learned about operating systems if you...
- want to work on the above problems
- care about what's going on under the hood
- have to build high-performance systems
- need to diagnose bugs or security problems

Class structure

Lectures
- first, basic O/S ideas
- then extended inspection of xv6, a traditional O/S
- finally, a series of recent topics
- homework
Lab: JOS, a small O/S for x86 in an exokernel style
- you build it, six labs, final project of your choice
- kernel interface: expose hardware, but protect -- no abstractions!
- unprivileged library: fork, exec, pipe, ...
- applications: file system, shell, ..
- development environment: gcc, bochs
- lab 1 is out
Two quizzes
- mid-term (in class)
- final (during exam week)

Case study: the shell (simplified)

interactive command execution and a programming language
typically handles login session, runs other processes
look at some simple examples of shell operations, how they use different O/S abstractions, and how those abstractions fit together. See Unix paper if you are unfamiliar with the shell.
Final lab is a simple shell.

Basic structure:

	while (1) {
	    write (1, "$ ", 2);
	    readcommand (command, args);   // parse user input
	    if ((pid = fork ()) == 0) {  // child?
		exec (command, args, 0);
	    } else if (pid > 0) {   // parent?
		wait (0);   // wait for child to terminate
	    } else {
		perror ("Failed to fork\n");
	    }
	}

system calls: read, write, fork, exec, wait. conventions: -1 return value signals error, error code stored in errno, perror prints out a descriptive error message based on errno.
What's the shell doing? fork, exec, wait: process diagram (PID, address space -- memory of the process, parent links). fork returns twice, in some sense!
The split of process creation into fork and exec turns out to have been an inspired choice, though that might not have been clear at the time; see the assigned paper for today.
why call "wait"? to wait for the child to terminate and collect its exit status. (if child finishes, child becomes a zombie until parent calls wait.)
Example:
```
	$ ls
```
how does ls know which directory to look at?
how does it know what to do with its output?
I/O: process has file descriptors, numbered starting from 0.
system calls: open, read, write, close
numbering conventions:
- file descriptor 0 for input (e.g., keyboard). read_command:
```
     read (0, buf, bufsize)
```
- file descriptor 1 for output (e.g., terminal)
```
     write (1, "hello\n", strlen("hello\n"))
```
- file descriptor 2 for error (e.g., terminal)
on fork, child inherits open file descriptors from parent (show in process diagram).
on exec, process retains file descriptors, except those specifically marked as close-on-exec: fcntl(fd, F_SETFD, FD_CLOEXEC)
How does the shell implement:
```
     $ ls > tmp1
```
just before exec insert:
```
	close(1);
	creat("tmp1", 0666);   // fd will be 1
```
The kernel always uses the first free file descriptor, 1 in this case. Could use dup2() to clone a file descriptor to a new number.
Good illustration for why fork + exec vs. CreateProcess on Windows. (CreateProcess takes 24 arguments.)
What if you run the shell itself with redirection?
```
     $ sh < script > tmp1
```
If for example the file script contains
```
     echo one
     echo two
```
FD inheritance makes this work well.
What if we want to redirect multiple FDs (stdout, stderr) for programs that print to both?
```
    $ ls f1 f2 nonexistant-f3 > tmp1 2> tmp1
```
after creat, insert:
```
	close(2);
	creat("tmp1", 0666);   // fd will be 2
```
why is this bad? illustrate what's going on with file descriptors. better:
```
	close(2);
	dup(1);		       // fd will be 2
```
or in bourne shell syntax,
```
    $ ls f1 f2 nonexistant-f3 > tmp1 2>&1
```
Linux has a nice representation of a process and its FDs, under /proc/PID/
- maps: VA range, perms (p=private, s=shared), offset, dev, inode, pathname
- fd: symlinks to files pointed to by each fd. (what's missing in this representation?)
- can do fd manipulation in shell and see it reflected in /proc/$$/fd

how to run a series of programs on some data?

	$ sort < file.txt > tmp1
	$ uniq tmp1 > tmp2
	$ wc tmp2
	$ rm tmp1 tmp2

can be more concisely done as:

        $ sort < file.txt | uniq | wc

A pipe is a one-way communication channel. Here is a simple example:

        int fdarray[2];
        char buf[512];
        int n;

        pipe(fdarray);
        write(fdarray[1], "hello", 5);
        n = read(fdarray[0], buf, sizeof(buf));
        // buf[] now contains 'h', 'e', 'l', 'l', 'o'

file descriptors are inherited across fork(), so this also works:

        int fdarray[2];
        char buf[512];
        int n, pid;

        pipe(fdarray);
        pid = fork();
        if(pid > 0){
          write(fdarray[1], "hello", 5);
        } else {
          n = read(fdarray[0], buf, sizeof(buf));
        }

How does the shell implement pipelines (i.e., cmd 1 | cmd 2 |..)? We want to arrange that the output of cmd 1 is the input of cmd 2. The way to achieve this goal is to manipulate stdout and stdin.

The shell creates processes for each command in the pipeline, hooks up their stdin and stdout, and waits for the last process of the pipeline to exit. Here's a sketch of what the shell does, in the child process of the fork() we already have, to set up a pipe:

	    
	    int fdarray[2];

  	    if (pipe(fdarray) < 0) panic ("error");
	    if ((pid = fork ()) == 0) {  child (left end of pipe)
	       close (1);
	       tmp = dup (fdarray[1]);   // fdarray[1] is the write end, tmp will be 1
	       close (fdarray[0]);       // close read end
	       close (fdarray[1]);       // close fdarray[1]
	       exec (command1, args1, 0);
	    } else if (pid > 0) {        // parent (right end of pipe)
	       close (0);
	       tmp = dup (fdarray[0]);   // fdarray[0] is the read end, tmp will be 0
	       close (fdarray[0]);
	       close (fdarray[1]);       // close write end
	       exec (command2, args2, 0);
	    } else {
	       printf ("Unable to fork\n");
            }

Who waits for whom? (draw a tree of processes)
Why close read-end and write-end? ensure that every process starts with 3 file descriptors, and that reading from the pipe returns end of file after the first command exits.
How do you create a background job?
```
        $ compute &
```
How does the shell implement "&", backgrounding? (Don't call wait immediately).
More details in the shell lecture later in the term.