​

Networking & Network File System

CS-446/646

C. Papachristos

Robotic Workers (RoboWork) Lab

University of Nevada, Reno

Networking Overview

Networking Overview

​

Computer Networking

​

​

​

​

​

​

Goal

• Two Applications on different computers exchange Data

​

• Requires Inter-Node (not just Inter-Process) Communication

CS446/646 C. Papachristos

Networking Overview

Networking Overview

​

• 7-Layer model: Conceptual Network design
• 4-Layer model: Practical Internet implementation

​

​

​

​

​

CS446/646 C. Papachristos

… also UDP, SCTP

Networking Overview

CS446/646 C. Papachristos

Networking Overview

Data Link Layer – Indirect Connectivity

​

Role: Definition of Format of data (Packet) on the Network (Device MAC Addresses, Switches)

​

• When no Direct Physical Connection to Destination exists

​

• Can Hop through multiple Devices

​

• Allows Links and Devices to be shared for multiple purposes
• Must determine which bits are part of which�Messages intended for which Destinations

​

• Packet-Switched Networks
• Pack Bytes together intended for a Destination
• Append a Packet Header with intended Destination
• Due to Routing, delivery can happen Out-of-Order

​

​

CS446/646 C. Papachristos

Networking Overview

Data Link Layer : Ethernet

​

Role: Definition of Format of data (Packet) on the Network (Device MAC Addresses, Switches)

​

• Originally designed for shared medium (Coax), now generally not shared (Switched)

​

• Vendors give each Device a unique 48-bit MAC Address
• Helps specify which card (Device) should receive a Packet

​

• Ethernet Switches can scale to switch Local Area Networks (LAN) (thousands of Hosts), but not much larger

​

• Packet format:

​

• Preamble: Helps Device recognize start of Packet
• CRC: Allows receiving card to ignore corrupted Packets
• Body: Up to 1,500 Bytes for same Destination
• All other fields must be set by sender’s OS�(NIC tells OS what the card’s MAC Address is, Special Addresses used for Broadcast/Multicast)

​

​

​

​

CS446/646 C. Papachristos

Networking Overview

Network Layer : Internet Protocol (IP) Suite

​

Role: Handles Routing, Logical Addressing (Host IP Addresses, Routers)

​

• IP used to connect multiple Networks
• Runs over a variety of Physical Networks
• Hence can connect Ethernet, DSL, Mobile Networks, etc.
• Most computers today speak IP

​

• Every Host has a unique 4-Byte IP address (16-Bytes for IPv6)
• (Or at least thinks it has, when there is Address shortage)
• e.g. www.ietf.org → 104.20.0.85

​

• Packets are Routed based on Destination IP Address
• Address space is structured to make Routing practical at global scale
• e.g. 134.197.78.* goes to UNR
• Therefore Packets need both Host IP Addresses, as well as�Device MAC Addresses

​

​

​

​

CS446/646 C. Papachristos

Networking Overview

Transport Layer : UDP and TCP

​

Role: Management of End-to-End communication (TCP, UDP)

​

• UDP and TCP most popular Transport-Layer Protocols on IP
• Both use 32-bit (i.e. 4-Byte) IP Address, as well as a 16-bit Port Number
• Applications Bind a Port & receive traffic to that Port

​

UDP – User Datagram Protocol

• Exposes Packet-Switched nature of Internet
• Application may suffer from sent Packets being dropped, reordered, or duplicated�(but generally not corrupted, Remember: CRC)

​

TCP – Transmission Control Protocol

• Provides illusion of a reliable “pipe” between two Processes on two different Machines
• Masks lost & reordered Packets so Application doesn’t have to worry
• Handles Congestion & Flow Control

​

​

​

CS446/646 C. Papachristos

Networking Overview

Fundamental Principles: Packet-Switching & Layering

​

Packet-Switching

• A Packet is a self-contained unit of Data which contains information necessary for it to reach its intended Destination
• Independently, for each arriving Packet, compute its Outgoing Link
• Makes forwarding simple (depends only on information within Packet)

​

Layering

• Break up system functionality into a hierarchy of Layers
• Each Layer uses only the service of the Layer below it in the hierarchy
• Layers communicate sequentially with the Layers above or below them

​

​

CS446/646 C. Papachristos

Networking Overview

Fundamental Principle: Encapsulation

​

• Essentially, sticking Packets inside Packets

​

• How Packet-Switching & Layering are realized in a system
• e.g. an Ethernet Frame may encapsulate an IP Packet
• An IP Router forwards an Ethernet Frame from one�Ethernet to another, creating a new Ethernet Frame�containing the same IP Packet
• In principle, an inner Layer should not depend on�outer Layers (not always true)

CS446/646 C. Papachristos

Note:�Term “Packet” is somewhat loosely used sometimes.�More formal term:�

• Protocol Data Unit (PDU):�A single unit of information transmitted among peer entities of a Network; composed of:�a) Protocol-specific Control Data�b) User Data

TCP PDU: “Segment”

IP PDU: “Packet”

Ethernet PDU: “Frame”

App PDU: “Message”

Systems Issues

TCP: Unreliability of IP

​

• Network does not deliver Packets reliably
• May drop Packets, reorder Packets, delay Packets
• May even corrupt Packets, or duplicate them
• Implementation requirements for a reliable Transport Protocol (TCP) on top of an IP Network?

​

Note: This is entirely handled by an OS that runs at the End-Nodes

​

Naïve Approach: Wait for Ack for each Packet

• Send a Packet, wait for Acknowledgment, send next Packet
• If no Ack, Timeout and try again

​

Problems?

• Low performance over high-delay Network�(Bandwidth is one Packet per Round-Trip Time)
• Possible “Congestive Collapse” of Network�(e.g. everyone keeps retransmitting when Network is overloaded)

CS446/646 C. Papachristos

Systems Issues

TCP: Performance Limitations & Bandwidth-Delay

​

• Network delay over Wide Area Network (WAN) will never improve much
• But Throughput (bps) is constantly improving

​

• Can view Network as a “pipe” :

​

• For full Utilization want # Bytes in flight ≥ Bandwidth × Delay�(But don’t want to overload the Network, either)

​

• What if Protocol doesn’t involve Bulk Transfer?
• e.g. a “ping-pong” Protocol will have poor Throughput

​

• Additional efficiency considerations for Protocol implementation:
• Concurrency & Response Time also critical for good Network utilization

CS446/646 C. Papachristos

Systems Issues

CS446/646 C. Papachristos

Note:�Term “TCP Packet” is both informal and formal usage. In more precise terminology “Segment” refers to the TCP Protocol Data Unit (PDU), “Packet�/Datagram” to the IP PDU, and “Frame” to the Data Link Layer PDU.

UDP uses “Datagrams” as its PDU for connectionless communication.

Systems Issues

TCP: Implementation OS issues

​

• Have to track Un-Ack(nowledg)-ed Data
• Sender keeps a copy around until Receiver Acknowledges it
• Sender keeps a Timer around to retransmit if no Ack
• Receiver must keep out-of-order Segment & reassemble

​

• When to wakeup receiving Process (i.e. “push” TCP-received Data to Application Layer) ?
• e.g. Sender calls write(fd, message, 8000); (Note: For Sockets, identical to send() w/o flags)
• First TCP Segment arrives, but is only 512 Bytes
• Receiving Host could wake receiving Process, but useless without full Message
• TCP Sender sets PSH (“push”) flag of TCP Header of end Segment of 8000-Byte write data

​

• When to immediately send short Segment –vs– Wait for more Data
• “Nagle’s Algorithm”: To reduce number of small Packets on wire, usually allow sending only one Un-Ack-ed short Segment, i.e. wait until previous packets are Ack’ed and in the meantime accumulate Data in Send Buffer
• But bad for some Apps, so provide NODELAY option

​

• OS must Ack received Segments very quickly
• Otherwise, effectively increases Round Trip Time, decreasing Bandwidth

C. Papachristos

OS Networking Facilities

CS446/646 C. Papachristos

OS Networking Facilities

Socket Naming

​

• TCP & UDP name Communication Endpoints by using:
• 32-bit (i.e. 4 Byte) IPv4 Address specifies Machine (128-bits for IPv6 Address)
• 16-bit TCP/UDP Port number that demultiplexes Endpoints within a Host

​

• For TCP, a “Connection” is thus defined by a “5-Tuple”:
• 1) Protocol (TCP/UDP), 2) Local IP, 3) Local Port, 4) Remote IP, 5) Remote Port
• “Connected” Sockets are required to implement TCP (also, Connected Mode UDP exists)

​

• OS keeps track of the Connection and its State via a Protocol Control Block (PCB) structure
• Keep all PCBs in a Hash Table
• When Packet arrives (if Destination IP Address belongs to Host), use its 5-Tuple to�Lookup the corresponding PCB and determine what to do with Packet

CS446/646 C. Papachristos

OS Networking Facilities

System Calls for using TCP

​

Client

​

​

​

​

​

Create a Socket – Obtain a Socket File Descriptor

sockfd = socket(domain, type, protocol);

​

Assign it a (Protocol-specific sockaddr* ) Address *

bind(sockfd, sockaddr, addrlen);

​

Connect to listening Socket

connect(sockfd, sockaddr, addrlen);

​

* This call to bind() is optional; connect() can bind to all�Interfaces and pick some high-numbered Port.�Used if Server restricts Clients to use specific Port (/range), or if�Client with multiple NICs wishes to use specific Address for its traffic.

CS446/646 C. Papachristos

Server

Create a Socket – Obtain a Socket File Descriptor

sockfd = socket(domain, type, protocol);

​

Assign it a (Protocol-specific sockaddr* ) Address

bind(sockfd, sockaddr, addrlen);

​

Mark Socket as passive – i.e. listening for Clients

listen(sockfd, backlog);

​

Accept Connection (can Block if none pending on Queue)

accept(sockfd, sockaddr, addrlen);

…

…

…

… return, Socket ESTABLISHED

Note: Using sockaddr* allows different types of Proto-col-specific structs to be passed (start of struct is Protocol family type information (sa_family_t), rest is Protocol-specific data (char sa_data[])

OS Networking Facilities

System Calls for using UDP

​

• Use socket() System Call with type = SOCK_DGRAM; then bind() as before

​

New System Calls for sending individual Packets:

​

• numbytes = sendto(sockfd, msgbuf, len, flags, sockaddr_dest, addrlen);

​

• numbytes = recvfrom(sockfd, msgbuf, len, flags, sockaddr_from, addrlen);

​

Note: Must send/get Peer Address with each Packet

​

Alternatively:

• Can also use UDP in Connected Mode
• connect() assigns remote Address
• Use send() / recv() System Calls
• like sendto() /recvfrom() without last 2 arguments

CS446/646 C. Papachristos

OS Networking Facilities

Uses of Connected Mode UDP Sockets

​

• Kernel demultiplexes Packets based on Port
• Allows different Processes to get Packets from different Peers
• For security, Ports [0, 1023] are considered Privileged (e.g. Port 80 – for HTTP)
• Usually can’t be bound by non-root Processes
• Unless e.g. the Process is granted the CAP_NET_BIND_SERVICE Capability
• So, services have to be initiated by root, but can then safely inherit �Privileged UDP Port that is already Connected (Connected Mode) to a particular Peer

​

• Feedback based on Internet Control Message Protocol (ICMP) Messages
• Assuming no Process has successfully bound the UDP Port you sent Packet to:
• With sendto(), you might think Network is dropping Packets
• Server sends back ICMP “Port Unreachable” Message, but can only detect it when using Connected Mode UDP Sockets

CS446/646 C. Papachristos

Implementing Networking in the Kernel

Sockets Implementation (Part I)

​

• Need to implement Layering efficiently
• Add UDP Header to Message, Add IP Header to UDP Datagram, etc.
• De-encapsulate Ethernet Packet so IP-handling code doesn’t get confused by Ethernet Header

​

• Don’t want to store entire Packets contiguously in Memory
• Moving Data to make room for new Header would be slow

​

• BSD solution: mbufs [Leffler] (Note: [Leffler] calls m_nextpkt by old name m_act)
• Small, fixed-size (256-Byte) structs
• Makes allocation/deallocation easy (no Fragmentation)

​

• BSD mbufs (“Message Buffers”) working example in this Lecture
• Linux uses sk_buffs (“Socket Buffers”); similar idea�(https://linux-kernel-labs.github.io/refs/pull/189/merge/labs/networking.html#linux-networking)

CS446/646 C. Papachristos

Implementing Networking in the Kernel

mbuf Details

​

• A Packet is made up of multiple mbufs
• Chained together by m_next
• Such linked mbufs called Chains

​

• Chains linked with m_nextpkt
• Linked Chains known as Queues
• e.g. Device Output Queue

​

• Total mbuf size 256 Bytes ⇒ ∼230 Bytes of Data – in m_dat�(depends on size of Pointers)
• First mbuf in Chain (“Head” of Chain) holds the Packet Header

​

• Cluster mbufs have more Data
• ext Header points to Data
• Up to 2 KB non-collocated with mbuf
• m_dat unused in this case

• m_flags is bitwise OR�of various bits
• e.g. if Cluster, or if�Packet Header used

C. Papachristos

Implementing Networking in the Kernel

Adding/Deleting Data with mbufs

​

• mbuf.m_data is used to point to the start of Data of that mbuf
• Can either be m_dat, or ext.buf for a Cluster mbuf
• But also: Can even point into middle of that area (!)
• Flexible handling of stripping away a front chunk from the Data

​

• To strip off a Packet Header (e.g. TCP/IP)
• Increment m_data, decrement m_len

​

• To strip off end of Packet
• Decrement m_len

​

• Can add more data to an mbuf if its Data buffer is not already full
• Otherwise, can append that data to the Chain
• Chain-in a new mbuf at Head/Tail of existing Chain

CS446/646 C. Papachristos

Implementing Networking in the Kernel

mbuf Utility Functions

​

mbuf * m_copym(mbuf * m, int off, int len, int wait);

• Creates a copy of a subset of a mbuf Chain
• Doesn’t copy Clusters, just increments reference-counting (for efficiency)
• wait says what to do if no Memory (wait or return NULL)

​

void m_adj(struct mbuf * mp, int len);

• Trim len Bytes from Head or (if negative) Tail of Chain

​

mbuf * m_pullup(struct mbuf * m, int len);

• Put first len Bytes existing in a mbuf Chain (starting at m) contiguously into the Data area of the Head mbuf structure (returns the Head mbuf of the Chain – now altered)

​

• Example with Layering: An Ethernet Frame containing an IP Packet
• Trim Ethernet Header using m_adj()
• Call m_pullup(n, sizeof (ip_hdr));
• Access IP Header as regular C struct (now contiguous !)

CS446/646 C. Papachristos

Implementing Networking in the Kernel

Sockets Implementation (Part II)

​

Each struct socket holds associated data:

• Associated File Descriptor information, e.g in Linux contains a struct file * (back-Pointer to File)
• Send Buffer & Receive Buffer (e.g. in Linux has struct sock *sk, that contains a struct sk_buff *)
• Note 1: struct sk_buff also contains back-Pointer to the owning struct sock Socket
• Note 2: struct sock also contains back-Pointer to the owning struct socket Socket

​

• State, e.g in Linux a socket_state (SS_FREE, SS_UNCONNECTED, SS_CONNECTING, SS_CONNECTED, SS_DISCONNECTING)
• Wait Queues (e.g. for incoming Connections on a listen()-ing Socket)
• Protocol Control Block (PCB) information (e.g. in Linux sk->sk_protocol of type unsigned int)
• A Protocol Handler, i.e. struct protosw * (equivalently in Linux: struct proto_ops *)

​

• Remember: A Protocol Control Block (PCB) represents a way to Lookup Protocol-specific info�e.g. for TCP:

​

CS446/646 C. Papachristos

Note: In the Linux Kernel, struct socket is a higher-level data structure for Socket functionalities (e.g. for System Calls), and struct sock is an imple-mentation of Address Family AF_INET (IPv4 Protocol)

• A 5-Tuple (Protocol (TCP), Source IP Address & Port, Destination IP Address & Port)
• Information about received Packets & position in stream
• Information about Un-Ack-nowledged sent Packets
• Information about Timeouts
• Connection State information (for Setup/Teardown)

Implementing Networking in the Kernel

Protocol Handler: protosw Structure (Linux equivalent: proto_ops)

​

• Goal: Abstraction Interface for different Protocol Handlers
• In C++ we could use virtual functions with a generic Socket struct
• In C, protosw (https://web.mit.edu/freebsd/head/sys/sys/protosw.h) contains function Pointers

​

• Also includes a few data fields …
• type, domain, protocol: To match socket() args, to know which protosw to select
• flags: To specify important properties of Protocol such as…

​

• … some Protocol flags:
• PR_ATOMIC: Exchange atomic Messages only (like UDP, not TCP)
• PR_ADDR: Address given with Messages (like Unconnected UDP)
• PR_CONNREQUIRED: Requires Connection (like TCP)
• PR_WANTRCVD: Notify Socket of consumed Data (e.g. so TCP can wake-up a sending Process blocked by Flow Control)

CS446/646 C. Papachristos

Implementing Networking in the Kernel

protosw Interface Functions (Note: Function Pointers)

​

• pr_slowtimo – Called every 0.5 s for Timeout processing

​

• pr_drain – Called when system low on space

​

• pr_input – Returns mbuf Chain of Data read from Socket

​

• pr_output – Takes mbuf Chain of Data written to Socket

​

• pr_usrreq – Multi-purpose user-request hook
• Used for bind/listen/accept/connect/disconnect operations
• Used for out-of-band Data

CS446/646 C. Papachristos

Implementing Networking in the Kernel

Network Interface Cards (NICs)

​

• Each NIC Driver needs to provide a struct ifnet implementation
• Note: struct ifnet is just an Abstraction Interface (like struct protosw)

​

• Data fields:
• Interface Name (e.g. "eth0")
• Address List via struct ifaddr (e.g. Ethernet Address, Broadcast Address, Netmask, ...)
• Information for its Layer 2 Protocol via if_l2com (includes Address Resolution Protocol (ARP) info)
• Maximum Packet Size
• Send Queue

​

• Interface Functions (Note: Function Pointers):
• if_input – To dequeue Packet and strip off Header
• if_output – To prepend Header and enqueue Packet
• if_transmit – To transmit a Packet or enqueue it (if the Interface is in use)
• if_start – To start transmitting Queued Packets
• Also if_ioctl, if_init, if_resolvemulti

CS446/646 C. Papachristos

Implementing Networking in the Kernel

NIC Input Handling

​

​

​

​

​

​

1) NIC Driver figures out Protocol from incoming Packet

​

2) Enqueue Packet for appropriate Protocol Handler

• If Queue full, drop Packet (can create “Livelock” [Mogul])

​

3) Posts “Soft Interrupt” for Protocol-Layer processing

• Runs at lower Priority than Hardware (NIC) Interrupt
• … but at higher Priority than Process-Context Kernel code

CS446/646 C. Papachristos

Implementing Networking in the Kernel

Additional OS Responsibility: Routing

​

• In Packet-Switched Networks, the OS must Route all transmitted Packets
• Pertinent issue: Destination Machine may have multiple NICs plus loopback Interface. Which Interface should a Packet be sent to, i.e. what MAC Address should Ethernet Frame have?

Note: Addressing (requires IP & MAC Address – discovered through ARP) is not the same as Routing

​

• “Routing” : Determining the “next-hop” purely based on the Destination IP Address�(in the IP Header of the Packet), even if Host has multiple NICs with different MAC Addresses
• OSI model adds a Layer to allow for path discovery to the next Gateway; this Layer is responsible for Routing (but knows nothing about the MAC Address)
• OS maintains “Routing Table” (Static/Dynamic): Maps IP Address & Prefix-length → Next Hop
• Uses Radix Tree for efficient Lookup:

​

• At each hop, we get next-hop’s IP Address from Routing Table, and use its MAC Address
• Most OSes provide Packet Forwarding
• Received Packets with a non-local Destination Address get�Routed out another Interface

CS446/646 C. Papachristos

• Branch at each node in Tree based on single bit of Target
• When a leaf node is reached, that is your Next Hop

Network File Systems

Network File System (NFS)

​

• Basic Idea: Look like a Filesystem (e.g. FFS) to Applications
• But Data potentially stored on different Machine …
• … so reads and writes have to rely on Network transactions
• Also called “Distributed Filesystems”

​

Advantages:

• Easy to Share if Files available on multiple Machines
• Often easier to administer Servers than Clients
• Access way more data than fits on your local Disk
• Network + Remote-buffer Cache can be faster than local Disk

​

Disadvantages

• Network + Remote Disk slower than local Disk
• Network or Server may fail even when Client OK
• Complexity, Security issues

CS446/646 C. Papachristos

Network File Systems

NFS version 2 [Sandberg]

​

• Background: Sun Network Disk (ND) Protocol (specified in [RFC_1050])
• Creates Disk-like Device even on Diskless Workstations
• Can create a regular (e.g. FFS) Filesystem on it
• But no File Sharing – Why?
• FFS assumes Disk doesn’t change under it

​

• ND idea still used today by Linux Network Block Device (NBD)
• Useful for Network Booting/Diskless Machines
• But not for File Sharing

​

• Network File System (NFS) – goals (NFS v2 Protocol specified in [RFC_1094])
• File Sharing: Access same Filesystem from multiple Machines simultaneously
• Maintain Unix semantics
• Crash recovery
• Competitive performance with ND Protocol

CS446/646 C. Papachristos

Network File Systems

NFS Implementation

​

• Via struct vnode (OpenBSD) of OS Virtual File System (VFS) Layer
• Remember: Abstraction Interface via Function Pointers, i.e. poor man’s C++ (like struct protosw)

​

• vnode Structure abstractly represents a Filesystem object (File, Directory, Symlink, etc.)

​

• Interface defines generic set of “vnode operations” (i.e. contains Function Pointers):
• lookup, create, open, close, getattr, setattr, read, write, fsync, remove, link, rename, mkdir, rmdir, symlink, readdir, readlink, …
• Called through Function Pointers, so most System Calls don’t care what type of Filesystem�a File resides on

​

• NFS implementations of vnode operations perform Remote Procedure Calls (RPC)
• Client sends Request to Server over Network, awaits Response
• Each Filesystem-related System Call may require the execution of a series of RPCs
• System mostly determined by RPC [RFC_1831] Protocol
• Uses eXternal Data Representation (XDR) language [RFC_1832]

C. Papachristos

Network File Systems

NFS Stateless Operation

​

Designed for “Stateless Operation”

• i.e. Server does not keep track of Client Sessions (States of Connections) or File Locks
• Motivated by performance reasons, and need to recover from Server Crashes

​

• Requests are self-contained (not relying on the data of each other)

​

• Requests are mostly* Idempotent (no matter how many times executed, result is the same)
• Unreliable UDP Transport
• Client retransmits Requests until it gets a reply
• Writes must be “Stable” before Server returns

​

* Why mostly?

• Of course, Filesystem is not stateless – It stores Files
• e.g. mkdir can’t be Idempotent – 2^nd time performed, the Directory already exists
• But many operations, e.g. read, write are Idempotent

CS446/646 C. Papachristos

Network File Systems

NFS version 3

​

• Stateless, and same general architecture as NFS v2 (Note: vs NFS v4 which is Stateful)

​

• Specified in [RFC_1813] (subset of Open Group spec)
• XDR defines C structures that can be sent over Network; includes “Discriminated” unions (i.e. “tagged” to know which union field is active)
• Protocol essentially defined as a set of Remote Procedure Calls (RPCs)

​

• New RPC for Access
• Supports Clients and Servers with different uids/gids

​

• Better support for Caching
• Server can return from “Unstable” (UNSTABLE) write (while data still Cached at Client)
• More information for Cache Consistency

​

• Better support for EXCLUSIVE File CREATE operation
• Procol-level Atomicity of creation of File, i.e. without Locks (more later…)
• Remember: Protocol is “Stateless”

​

CS446/646 C. Papachristos

Network File Systems

Prelude: NFS v3 File Handles

​

​

​

• Server assigns an opaque File Handle to each File
• Client obtains 1^st File Handle “Out-Of-Band (OOB)” – Separate Mount Protocol
• File Handle hard to guess – Security enforced at Mount Time
• But any subsequent File Handles obtained through Lookups

​

• File Handle internally specifies Filesystem & File
• Device number, i-number, generation number, ...
• Generation number changes when inode recycled

​

• File Handle generally doesn’t contain Filename
• Clients may keep accessing an open File even after it has been renamed

CS446/646 C. Papachristos

struct nfs_fh3 {

/* XDR notation for variable-length array with 0-64 opaque bytes: */

opaque data<NFS3_FHSIZE>; /* NFS3_FHSIZE defined as 64 */

};

Network File Systems

Prelude: NFS v3 File Attributes

​

• Most operations can optionally return fattr3
• Contains Attributes used for Cache Consistency

CS446/646 C. Papachristos

struct fattr3 {

ftype3 type;

uint32 mode;

uint32 nlink;

uint32 uid;

uint32 gid;

uint64 size;

uint64 used;

specdata3 rdev;

uint64 fsid;

uint64 fileid;

nfstime3 atime;

nfstime3 mtime;

nfstime3 ctime;

};

struct post_op_attr {

bool_t attributes_follow;

union {

fattr3 attributes;

} post_op_attr_u;

};

Note: An XDR Discriminated union (RPC Data Description Language)

union post_op_attr switch (bool attributes_follow) {

case TRUE:

fattr3 attributes;

case FALSE:

void;

};

Network File Systems

NFS v3 Lookup

​

• RPC Procedure: LOOKUP3res NFSPROC3_LOOKUP(LOOKUP3args);

​

​

​

​

​

​

​

• Maps < Directory Handle (dir) , Filename (name) > → File Handle (object)
• Client walks Hierarchy one File at a time
• No Symlinks or Filesystem boundaries crossed
• If Symlink encountered, Client must expand it

CS446/646 C. Papachristos

struct diropargs3 {

nfs_fh3 dir;

filename3 name;

};

struct LOOKUP3resok {

nfs_fh3 object;

post_op_attr obj_attributes;

post_op_attr dir_attributes;

};

union LOOKUP3res switch (nfsstat3 status) {

case NFS3_OK:

LOOKUP3resok resok;

default:

LOOKUP3resfail resfail;

};

struct LOOKUP3resfail {

post_op_attr dir_attributes;

};

struct LOOKUP3args {

diropargs3 what;

};

Need this:

Network File Systems

NFS v3 Create

​

• RPC Procedure: CREATE3res NFSPROC3_CREATE(CREATE3args);

​

​

​

​

​

​

• UNCHECKED – Succeed even if duplicate File exists
• GUARDED – Check if duplicate File exists; Fail in this case
• EXCLUSIVE – Persistent record of CREATE (instead of Client�providing just Attributes, it directly provides a Verifier that is recorded in file�attributes; if same Client re-attempts EXCLUSIVE CREATE, Server returns�success, if another Client attempts EXCLUSIVE CREATE around same time,�its Verifier won’t match what’s recorded in file, Server returns NFS3ERR_EXIST)

union createhow3 switch (createmode3 mode) {

case UNCHECKED:

case GUARDED:

sattr3 obj_attributes;

case EXCLUSIVE:

createverf3 verf;

};

struct CREATE3args {

diropargs3 where;

createhow3 how;

};

enum createmode3 {

UNCHECKED = 0,

GUARDED = 1,

EXCLUSIVE = 2

};

Need this

too:

C. Papachristos

Network File Systems

NFS v3 Read

​

• RPC Procedure: READ3res NFSPROC3_READ(READ3args);

​

​

​

​

​

​

​

​

• offset explicitly specified (not implicit in File Handle – Remember: Stateless Design)
• Client can Cache result

CS446/646 C. Papachristos

struct READ3args {

nfs_fh3 file;

uint64 offset;

uint32 count;

};

union READ3res switch (nfsstat3 status) {

case NFS3_OK:

READ3resok resok;

default:

READ3resfail resfail;

};

struct READ3resok {

post_op_attr file_attributes;

uint32 count;

bool eof;

opaque data<>;

};

struct READ3resfail {

post_op_attr file_attributes;

};

Network File Systems

NFS v3 Data Caching

​

• Client can Cache File Blocks of Data read and written previously, and use them in a Cache Consistency- based approach by using times in fattr3
• nfstime3 ctime: Time of last change to inode
• nfstime3 mtime: Time of last modification to File�(Changed by explicitly setting mtime, increasing size of File, changing Permissions, etc.)

​

Data Caching Algorithm:

• If our Cached values for ctime or mtime appear to have now changed based on the contents of the last received fattr3 (e.g. due to another Client’s actions), flush any Cached File Blocks

CS446/646 C. Papachristos

Network File Systems

C. Papachristos

Network File Systems

NFS v2 Write

​

• RPC Procedure: attrstat NFSPROC_WRITE(writeargs);

​

​

​

​

​

​

​

​

• On successful WRITE RPC, returns new File Attributes (as fattr)
• Can NFS v2 keep Cached copy of the File’s Blocks after �a successful WRITE RPC?

struct writeargs {

fhandle file;

unsigned beginoffset;

unsigned offset;

unsigned totalcount;

opaque data<NFS_MAXDATA>;

};

union attrstat switch (stat status) {

case NFS_OK:

fattr attributes;

default:

void;

};

Need this:

CS446/646 C. Papachristos

Network File Systems

NFS v2 Write Race Condition

​

​

​

​

​

​

​

• Suppose Client overwrites a 2-Block File sequentially
• Client A knows File Attributes after write() Block 1 & after write() Block 2
• Reported back from WRITE RPC
• But Client B could overwrite Block 1 inbetween
• No way for Client A to know this hasn’t happened (based on what the RPCs report)
• Must flush Cache before next File read() (or at least open())

C. Papachristos

Report back mtime due to A’s write A1

Report back mtime due to A’s write A2

Report back mtime due to B’s write B1

Network File Systems

NFS v3 Write

​

• RPC Procedure: WRITE3res NFSPROC3_WRITE(WRITE3args);

​

​

​

​

​

​

Two goals for NFS v3 write:

• Don’t force Clients to flush Cache after WRITEs
• Don’t equate Cache Consistency with Crash Consistency
• i.e. don’t wait for Disk just so�another Client can see Data

​

struct WRITE3args {

nfs_fh3 file;

uint64 offset;

uint32 count;

stable_how stable;

opaque data<>;

};

enum stable_how {

UNSTABLE = 0,

DATA_SYNC = 1,

FILE_SYNC = 2

};

union WRITE3res switch (nfsstat3 status)

{

case NFS3_OK:

WRITE3resok resok;

default:

WRITE3resfail resfail;

};

struct WRITE3resok {

wcc_data file_wcc;

count3 count;

stable_how committed;

writeverf3 verf;

};

struct WRITE3resfail {

wcc_data file_wcc;

};

Need this

too:

CS446/646 C. Papachristos

Network File Systems

NFS v3 Write Results

​

• RPC Procedure: WRITE3res NFSPROC3_WRITE(WRITE3args);

​

​

​

​

​

​

• Write Results
• Several fields added to achieve these goals

union WRITE3res switch (nfsstat3 status)

{

case NFS3_OK:

WRITE3resok resok;

default:

WRITE3resfail resfail;

};

struct WRITE3resok {

wcc_data file_wcc;

count3 count;

stable_how committed;

writeverf3 verf;

};

struct WRITE3resfail {

wcc_data file_wcc;

};

CS446/646 C. Papachristos

struct wcc_data {

wcc_attr * before;

post_op_attr after;

};

struct wcc_attr {

uint64 size;

nfstime3 mtime;

nfstime3 ctime;

};

Network File Systems

NFS v3 Data Caching after a Write

​

• WRITE RPC will change mtime/ctime of a File
• Field after (type: post_op_attr ) will contain new times
• Same as with NFS v2, but just relying on just this information alone required to flush the Cache

​

• With NFS v3, field before (type: wcc_attr ) additionally contains previous values before this WRITE RPC
• If before matches our Cached values, no other Client has gotten to change the File inbetween
• OK to update Attributes without flushing Data Cache

CS446/646 C. Papachristos

Network File Systems

NFS v3 Write Stability

​

• Server write must be at least as stable (type: stable_how ) as requested

​

• If Server returns write UNSTABLE:
• Means Permissions okay, enough free Disk space
• But Data not on Disk and might disappear (e.g. after a Crash)

​

• If Server returns DATA_SYNC:
• Data on Disk, maybe not Attributes

​

• If Server returns FILE_SYNC:
• Operation complete and stable

CS446/646 C. Papachristos

Network File Systems

NFS v3 Commit Operation

​

• Client cannot discard any UNSTABLE write
• If Server crashes, Data will be lost

​

• RPC: COMMIT3res NFSPROC3_COMMIT(COMMIT3args);
• Provides a data-sync mechanism to be used with asynchronous WRITE operations

​

• NSFPROC3_COMMIT RPC “commits” a range of a File to Disk
• Invoked by Client when Client cleaning buffer Cache
• Invoked by Client when User closes/flushes a File

​

• How does Client know if Server crashed?
• WRITE and COMMIT RPCs return: writeverf3 WRITE3res.verf
• Value changes after each Server Crash (e.g. can be the Boot Time)
• Client must resend all UNSTABLE writes if verf value changes

CS446/646 C. Papachristos

struct COMMIT3args {

nfs_fh3 file;

uint64 offset;

uint32 count;

};

Network File Systems

NFS v3 Attribute Caching

​

• NFS has no OPEN or CLOSE operations (i.e. RPCs)

​

• “Close-to-Open” (CTO) Cache Consistency
• Client should not maintain a Cached File version after close() (e.g to re-open())
• Client responsible to WRITE/COMMIT any changes on close()
• Client responsible to GETATTR/ACCESS on open()s to fetch Attributes from Server

​

• However, we can have lots of other needs for Cached Attributes (e.g. ls -al)
• Attributes Cached between 5 and 60 seconds
• Files recently changed more likely to change again
• Do Weighted Cache Expiration based on age of File

​

• Drawbacks of combined a) CTO Cache Consistency & b) Attribute Caching:
• Must pay for Round-Trip to Server on every File open()
• Can get stale info when stat-ting a File (CTO does not guarantee consistency for Data changed while File already open)

Note: No OPEN/CLOSE�RPCs, NFS is Stateless

CS446/646 C. Papachristos

Time for Questions !

CS-446/646

CS446/646 C. Papachristos