trending/coursebook
1
0
Fork 0
mirror of https://github.com/cs341-illinois/coursebook.git synced 2026-10-02 08:04:38 +08:00
Code Issues Packages Projects Releases Wiki Activity

Updated trial run of the wikibook project

Browse Source
This commit is contained in:
Bhuvan Venkatesh
2018-02-04 17:17:52 -06:00
parent 97b46b3d15
commit dc669fb882
19 changed files with 1274 additions and 1681 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Makefile
+2 -2
View File
@@ -1,5 +1,5 @@
# Find all tex files one directory down
TEX=$(shell find . -maxdepth 2 -mindepth 2 -path "./.git*" -prune -o -type f -iname "*.tex" -print)
TEX=$(shell find . -path "./.git*" -prune -o -type f -iname "*.tex" -print)
MAIN_TEX=main.tex
PDF_TEX=$(patsubst %.tex,%.pdf,$(MAIN_TEX))
BASE=$(patsubst %.tex,%,$(MAIN_TEX))
@@ -12,7 +12,7 @@ all: $(PDF_TEX)
$(PDF_TEX): $(TEX) $(MAIN_TEX) $(BIBS) Makefile
-latexmk latexmk -interaction=nonstopmode -quiet -f -pdf $(MAIN_TEX) 2>&1 >/dev/null
-latexmk -quiet -interaction=nonstopmode -f -pdf $(MAIN_TEX) 2>&1 >/dev/null
-@latexmk -c
-@rm *aux *bbl *glg *glo *gls *ist *latexmk *fls
appendix/appendix.tex
+41 -2
View File
@@ -242,12 +242,51 @@ Here, by using the \texttt{x} command with parameters \texttt{16xb}, we can see
\section{Life in the terminal}
\todo
\todo{Life in the Terminal}
\texttt{0x43\ 0x61\ 0x74\ 0xe0\ 0xf9\ 0xbf\ 0x5f\ 0xff\ 0x7f\ 0x00}
\section{Stack Smashing}\label{stack-smashing}
Each thread uses a stack memory. The stack `grows downwards' - if a function calls another function, then the stack is extended to smaller memory addresses. Stack memory includes non-static automatic (temporary) variables, parameter values and the return address. If a buffer is too small some data (e.g.~input values from the user), then there is a real possibility that other stack variables and even the return address will be overwritten. The precise layout of the stack's contents and order of the automatic variables is architecture and compiler dependent. However with a little investigative work we can learn how to deliberately smash the stack for a particular architecture.
The example below demonstrates how the return address is stored on the stack. For a particular 32 bit architecture \href{http://cs-education.github.io/sys/}{Live Linux Machine}, we determine that the return address is stored at an address two pointers (8 bytes) above the address of the automatic variable. The code deliberately changes the stack value so that when the input function returns, rather than continuing on inside the main method, it jumps to the exploit function instead.
\begin{code}[language=C]
// Overwrites the return address on the following machine:
// http://cs-education.github.io/sys/
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
void breakout() {
puts("Welcome. Have a shell...");
system("/bin/sh");
}
void input() {
void *p;
printf("Address of stack variable: %p\n", &p);
printf("Something that looks like a return address on stack: %p\n", *((&p)+2));
// Let's change it to point to the start of our sneaky function.
*((&p)+2) = breakout;
}
int main() {
printf("main() code starts at %p\n",main);
input();
while (1) {
puts("Hello");
sleep(1);
}
return 0;
}
\end{code}
There are \href{https://en.wikipedia.org/wiki/Stack_buffer_overflow}{a lot} of ways that computers tend to get around this.
\section{System Programming Jokes}
\texttt{0x43\ 0x61\ 0x74\ 0xe0\ 0xf9\ 0xbf\ 0x5f\ 0xff\ 0x7f\ 0x00}
Warning: Authors are not responsible for any neuro-apoptosis caused by these ``jokes.'' - Groaners are allowed.
\subsection{Light bulb jokes}
deadlock/deadlock.bib
+2
View File
@@ -8,6 +8,7 @@
publisher={Wiley India Pvt. Limited}
}
@article{coffman1971system,
title={System deadlocks},
author={Coffman, Edward G and Elphick, Melanie and Shoshani, Arie},
@@ -19,6 +20,7 @@
publisher={ACM}
}
@article{rice,
ISSN = {00029947},
URL = {http://www.jstor.org/stable/1990888},
deadlock/deadlock.html
+136
View File
@@ -0,0 +1,136 @@
<html>
<head>
<meta charset="UTF-8">
</head>
<body>
<h1 id="deadlock">Deadlock</h1>
<p>Deadlock is defined as when the system cannot make and forward progress. In a lot of systems, Deadlock is just avoided by ignore the entire concept <span class="citation"></span>. Have you heard about turn it on and off again? That is partly because of this. For products where the stakes are low when you deadlock (User Operating Systems, Phones), it may be more efficient not not keep track of all of the allocations in order to keep deadlock from happening. But in the cases where “failure is not an option” - Apollo 13, you need a system that tracks deadlock or better yet prevents it entirely. Take the Apollo 13 module. It may have not failed because of deadlock, but probably wouldn’t be good to restart the system on liftoff.</p>
<p>Mission critical operating systems need this guarentee formally because playing the odds with people’s lives isn’t a good idea. Okay so how do we do this? We model the problem. Even though it is a common statistical phrase that all models are wrong, the more accurate the model is to the system that we are working with the better chance that it’ll work better.</p>
<h2 id="resource-allocation-graphs">Resource Allocation Graphs</h2>
<p>[13]<span>r</span><span>.35</span></p>
<p><img src="deadlock/images/resource_allocation.png" alt="image" /></p>
<p>One such way is modeling the system with a resource allocationg graph. A resource allocation graph tracks which resource is held by which process and which process is waiting for a resource of a particular type. It is very powerful and simple tool to illustrate how interacting processes can deadlock. If a process is <em>using</em> a resource, an arrow is drawn from the resource node to the process node. If a process is <em>requesting</em> a resource, an arrow is drawn from the process node to the resource node.If there is a cycle in the Resource Allocation Graph and each resource in the cycle provides only one instance, then the processes will deadlock. For example, if process 1 holds resource A, process 2 holds resource B and process 1 is waiting for B and process 2 is waiting for A, then process 1 and 2 process will be deadlocked.</p>
<p>[12]<span>r</span><span>.35</span></p>
<p><img src="deadlock/images/colorful.png" alt="image" /></p>
<p>Consider the following resource allocation graph. Assume that the processes ask for exclusive access to the file. If you have a bunch of processes running and they request resources and the operating system ends up in this state, you deadlock! You may not see this because the operating system may <strong>preempt</strong> some processes breaking the cycle but there is still a change that your three lonely processes could deadlock. You can also make these kind of graphs with <code>make</code> and rule dependencies with our parmake MP for example.</p>
<h2 id="coffman-conditions">Coffman conditions</h2>
<p>There are four <em>necessary</em> and <em>sufficient</em> conditions for deadlock – meaning if these conditions hold then there is a non-zero probability that the system will deadlock at any given iteration. These are known as the Coffman conditions <span class="citation"></span>.</p>
<ul>
<li><p>Mutual Exclusion: no two processes can obtain a resource at the same time.</p></li>
<li><p>Circular Wait: there exists a cycle in the Resource Allocation Graph, or there exists a set of processes {P1,P2,…} such that P1 is waiting for resources held by P2, which is waiting for P3,…, which is waiting for P1.</p></li>
<li><p>Hold and Wait: a process once obtaining a resource does not let go.</p></li>
<li><p>No pre-emption: nothing can force the process to give up a resource.</p></li>
</ul>
<p>Deadlock can happen if and only if the four coffman conditions are satisified.</p>
<p><span class="math inline">→</span> If the system is deadlocked, the four coffman conditions are apparent.</p>
<ul>
<li><p>For the purposes of contradiction, assume that there is no cicular wait. If not then that means the resource wait graph is acyclic, meaning that there is at least one one process that is not waiting on any resources or it can grab a resource. Since the system can move forward, the system is not deadlocked.</p></li>
<li><p>For the purposes of contradiction, assume that there is no mutual exclusion. If not, that means that no process is waiting on any other process for a resource. This breaks circular wait and the previous argument proves correctness.</p></li>
<li><p>For the purposes of contradiction, assume that processes don’t hold and wait but our system still deadlocks. Since we have circular wait from the first condition at least one process must be waiting on another process. If that and processes don’t hold and wait, that means one process must let go of a resource. Since the system has moved forward, it cannot be deadlocked.</p></li>
<li><p>For the purposes of contradiction, assume that we have preemption, but the system cannot be un-deadlocked. Have one process, or create one process, that recognizes the circular wait that must be apprent from above and break on the of the links. By the first branch, we must not have deadlock.</p></li>
</ul>
<p><span class="math inline">←</span> If the four conditions are apparent, the system is deadlocked. We will prove that if the system is not deadlocked, the four conditions are not apparent. Though this proof is not formal, let us build a system with the three requirements not including circular wait. Let assume that there is a set of processes <span class="math inline"><em>P</em> = {<em>p</em><sub>1</sub>, <em>p</em><sub>2</sub>, ..., <em>p</em><sub><em>n</em></sub>}</span> and there is a set of resources <span class="math inline"><em>R</em> = {<em>r</em><sub>1</sub>, <em>r</em><sub>2</sub>, ..., <em>r</em><sub><em>m</em></sub>}</span>. For simplicity, a process can only request one resource at a time but the proof can be generalized to multiple. Let assume that the system is at different states at different times <span class="math inline"><em>t</em></span>. Let us assume that the state of the system is a tuple <span class="math inline">(<em>h</em><sub><em>t</em></sub>, <em>w</em><sub><em>t</em></sub>)</span> where there are two function <span class="math inline"><em>h</em><sub><em>t</em></sub> : <em>R</em> → <em>P</em> ∪ {unassigned}</span> that maps resources to the processes that own them (this is a function, meaning that we have mutual exclusion) and or unassigned and <span class="math inline"><em>w</em><sub><em>t</em></sub> : <em>P</em> → <em>R</em> ∪ {satisfied}</span> that maps the requests that each process makes to a resource or if the process is satisfied. Let <span class="math inline"><em>L</em><sub><em>t</em></sub> ⊆ <em>P</em> × <em>R</em></span> be a set of list of requests that a process uses to release a resource at any given time. The evolution of the system is at each step at every time.</p>
<ul>
<li><p>Release all resources in <span class="math inline"><em>L</em><sub><em>t</em></sub></span> the a process requests to resource.</p></li>
<li><p>Find a process that is requesting a resource</p></li>
<li><p>If that resource is available give it to that process, generating a new <span class="math inline">(<em>h</em><sub><em>t</em></sub>, <em>w</em><sub><em>t</em></sub>)</span> and exit the current iteration.</p></li>
<li><p>Else find another process and try the above if.</p></li>
</ul>
<p>If all processes have been surveyed and none updates the system, consider it deadlocked. More formally, this system is deadlocked means if <span class="math inline">∃<em>t</em><sub>0</sub>, ∀<em>t</em> ≥ <em>t</em><sub>0</sub>, ∀<em>p</em> ∈ <em>P</em>, <em>w</em><sub><em>t</em></sub>(<em>p</em>)≠satisfied and ∃<em>q</em>, <em>q</em> ≠ <em>p</em> → <em>h</em><sub><em>t</em></sub>(<em>w</em><sub><em>t</em></sub>(<em>p</em>)) = <em>q</em></span> (which is what we need to prove).</p>
<p>These conditions imply deadlock. Deadlock for a system is defined as no work can be done now or later. Work can be done if a process is satisfied, or we can release a resource to a process. No process is satisfied by definition. Since the system can’t preempt and release resources (by no preemption), the processes have to do it themselves. If the processes has a resource, it will not let it go until after it is satisfied by hold and wait. All resources requested by the processes are owned by other processes meaning that no process will let any resource go. Since we have shown no processes will give up a resource and no process is satisfied, the system is in deadlock.</p>
<p>The last condition to address is circular wait. Circular wait means that there exists <span class="math inline">∀<em>p</em> ∈ <em>P</em>, <em>w</em><sub><em>t</em></sub>(<em>p</em>)≠satisfied and ∃<em>q</em>, <em>q</em> ≠ <em>p</em> → <em>h</em><sub><em>t</em></sub>(<em>w</em><sub><em>t</em></sub>(<em>p</em>)) = <em>q</em></span>. Which is what we needed to show.</p>
<p>If you break any of them, you cannot have deadlock! Consider the scenario where two students need to write both pen and paper and there is only one of each. Breaking mutual exclusion means that the students share the pen and paper. Breaking circular wait could be that the students agree to grab the pen then the paper. As a proof by contradiction, say that deadlock occurs under the rule and the conditions. Without loss of generality, that means a student would have to be waiting on a pen while holding the paper and the other waiting on a pen and holding the paper. We have contradicted ourselves because one student grabbed the paper without grabbing the pen, so deadlock must not be able to occur. Breaking hold and wait could be that the students try to get the pen and then the paper and if a student fails to grab the paper then they release the pen. This introduces a new problem called <em>livelock</em> which will be discussed latter. Breaking preemption means that if the two students are in deadlock the teacher can come in and break up the deadlock by giving one of the students one the held on items or tell both students to put the items down.</p>
<p>Livelock relates to deadlock but it is not exactly deadlock. Consider the breaking hold and wait solution as above. Though deadlock is avoided, if we pick up the same device (a pen or the paper) again and again in the exact same pattern, neither of us will get any writing done. More generally, livelock happens when the process looks like it is executing but no meaningful work is done. Livelock is generally harder to detect because the processes generally look like they are working to the outside operating system wheras in deadlock the operating system generally knows when two processes are waiting on a system wide resource. Another problem is that there are nessisary conditions for livelock (i.e. deadlock does not occur) but not sufficient conditions – meaning there is no set of rules where livelock has to occur. You must formally prove in a system by what is known as an invariant. One has to enumerate each of the steps of a system and if each of the steps eventually (after some finite number of steps) leads to forward progress, the system is not livelocked. There are even better systems that prove bounded waits; a system can only be livelocked for at most <span class="math inline"><em>n</em></span> cycles which may be important for something like stock exchanges.</p>
<h2 id="approaches-to-solving-deadlock">Approaches to solving deadlock</h2>
<p>Ignoring deadlock is the most obvious approach that started the chapter out detailing. Quite humorously, the name for this approach is called the ostrich algorithm. Though there is no apparent source, the idea for the algorithm comes from the concept of an ostrich sticking its head in the sand. When the operating system detects deadlock, it does nothing out of the ordinary and hopes that the deadlock goes away. Now this is a slight misnomer because the operating system doesn’t do anything <em>abnormal</em> – it is not like an operating system deadlocks every few minutes because it runs  100 processes all requesting shared libraries. An operating system still preempts processes when stopping them for context switches. The operating system has the ability to interrupt any system call, potentially breaking a deadlock scenario. The OS also makes some files read-only thus making the resource shareable. What the algorithm refers to is that if there is an adversary that specifically crafts a program – or equivalently a user who poorly writes a program – that deadlock could not be caught by the operating system. For everyday life, this tends to be fine. When it is not we can turn to the following method.</p>
<p>Deadlock detection allows the system to enter a deadlocked state. After entering, the system uses the information that it has to break deadlock. As an example, consider multiple processes accessing files. The operating system is able to keep track of all of the files/resources through file descriptors at some level either abstracted through an API or directly. If the operating system detects a directed cycle in the operating system file descriptor table it may break one process’ hold through scheduling for example and let the system proceed. Why this is a popular choice in this realm is that there is no way of knowing which resources a program will select without running the program. This is an extension of Rice’s theorem <span class="citation"></span> that says that we cannot know any semantic feature without running the program (semantic meaning like what files it tries to open). So theoretically, it is sound. The problem then gets introduced that we could reach a livelock scenario if we preempt a set of resources again and again. The way around this is mostly probabilistic. The operating system chooses a random resource to break hold and wait. Now even though a user can craft a program where breaking hold and wait on each resource will result in a livelock, this doesn’t happen as often on machines that run programs in practice or the livelock that does happen happens for a couple of cycles. These kind of systems are good for products that need to maintain a non-deadlocked state but can tolerate a small chance of livelock for a short period of time. The following proof <strong>is not required for our 241 related puproses but is included for concreteness</strong>.</p>
<p>That livelock terminates with high probability. This meaning that for any probability level <span class="math inline"><em>l</em></span> we can produce a number of iterations <span class="math inline"><em>n</em></span> that the probability that the system is not livelocked after that state is at least <span class="math inline"><em>l</em></span>.</p>
<p>Let <span class="math inline"><em>L</em> = {<em>p</em><sub>1</sub>, <em>p</em><sub>2</sub>, <em>p</em><sub>3</sub>, ...}</span> be an infinite set that has probabilities if we choose a resource that causes livelock in the <span class="math inline"><em>i</em></span>th iteration of the livelocked by breaking a random resource. Also let the set have the property that <span class="math inline">∀<em>i</em> &gt; 0, ∃<em>j</em> &gt; <em>i</em>, <em>p</em><sub><em>j</em></sub> &lt; 1</span> Meaning that for all elements after any given element that there is atleast one element int he future that has a probability of breaking deadlock. The probability that the system is not livelocked after <span class="math inline"><em>n</em></span> iterations is <br /><span class="math display">$$P(n) = 1 - \prod\limits_{i = 1}^{n}p_i$$</span><br /> Consider the new set <span class="math inline"><em>E</em> = {<em>s</em><sub>1</sub>, <em>s</em><sub>2</sub>, <em>s</em><sub>3</sub>, ...}</span> obtained by selecting all non-one elements. The set must be infinite by our sets property. It is easy to show that <br /><span class="math display">$$\prod\limits_{i = 1}^{n}p_i = \prod\limits_{i = 1}^{n}s_i$$</span><br /> And since <br /><span class="math display">$$P(n) = 1 - \prod\limits_{i = 1}^{n}s_i$$</span><br /> is a strictly monotonically decreasing function, it must pass the threshold for the probability level at some point. More rigorously <br /><span class="math display">$$\begin{aligned}
Y =&amp; \sup E \\
\prod\limits_{i = 1}^{n}s_i &lt;&amp; Y^n \\
1 - \prod\limits_{i = 1}^{n}s_i &gt;&amp; 1 - Y^n \\
P(n) &gt;&amp; 1 - Y^n \\
P(n) &gt;&amp; l \\
1 - Y^n &gt;&amp; l \\
\log_Y(1 - l) &lt;&amp; n\end{aligned}$$</span><br /> We have found the number of steps in the set <span class="math inline"><em>E</em></span> and since our set has the property that gaps between non-one elements is finite, we can reconstruct the number of iterations by picking an <span class="math inline"><em>n</em></span> that satifies the <span class="math inline"><em>E</em></span> criteria and just adding the finite gaps together, which is what we needed to show.</p>
<p>Deadlock prevention is making sure that deadlock cannot happen, meaning that you break a Coffman condition. This works the best inside a single program and the software engineer making the choice to break a certain coffman condition. Consider the <a href="https://en.wikipedia.org/wiki/Banker&#39;s_algorithm">Banker’s Algorithm</a> <span class="citation"></span>. It is another algorithm for deadlock avoidance. The whole implementation is outside the scope of this class, just know that there are more generalized algorithms for operating systems.</p>
<p>The banker algorithm is actually not too complicated. We can start out with the single resource solution. Let’s say that I’m a banker. As a banker I have a finite amount of money. As having a finite amount of money, I want to make loans and eventually get my money back. Let’s say that we have a set of <span class="math inline"><em>n</em></span> people where each of them have a set amount or a limit <span class="math inline"><em>a</em><sub><em>i</em></sub></span> (<span class="math inline"><em>i</em></span> being the <span class="math inline"><em>i</em></span>th process) that they need to obtain before they can do any work. I keep track in my book how much I’ve given to each person <span class="math inline"><em>l</em><sub><em>i</em></sub></span>, and I have some amount of principle <span class="math inline"><em>p</em></span> at any given time. For people to request money, they do the following. Consider the state of the system <span class="math inline">(<em>A</em> = {<em>a</em><sub>1</sub>, <em>a</em><sub>2</sub>, ...},<em>L</em> = {<em>l</em><sub>1</sub>, <em>l</em><sub>2</sub>, ...},<em>p</em>)</span>. An assumption of this system is that we have <span class="math inline"><em>p</em> &gt; inf<em>a</em><sub><em>i</em></sub></span>, or we have enough money to satisfy one person. Also, each person will work for a finite period of time and give back our money.</p>
<ul>
<li><p>A person <span class="math inline"><em>j</em></span> requests <span class="math inline"><em>m</em></span> from me</p>
<ul>
<li><p>if <span class="math inline"><em>m</em> &lt; <em>p</em></span>, they are denied.</p></li>
<li><p>if <span class="math inline"><em>m</em> + <em>l</em><sub><em>j</em></sub> &gt; <em>a</em><sub><em>i</em></sub></span> they are denied</p></li>
<li><p>Pretend we are in a new state <span class="math inline">(<em>A</em> = {...,<em>a</em><sub><em>j</em></sub>, ...},<em>L</em> = {.., <em>l</em><sub><em>j</em></sub> + <em>m</em>, ...},<em>p</em> − <em>m</em>)</span> where the process is granted the resource.</p></li>
</ul></li>
<li><p>if now person <span class="math inline"><em>j</em></span> is either satisfied (<span class="math inline"><em>l</em><sub><em>j</em></sub> = =<em>a</em><sub><em>j</em></sub></span>) or <span class="math inline">∃<em>i</em>, <em>a</em><sub><em>j</em></sub> − <em>l</em><sub><em>j</em></sub> &lt; <em>p</em></span>. In other words we have enough money to satisfy one other person. If either, consider the transaction safe and give them the money.</p></li>
</ul>
<p>Why does this work? Well at the start we are in a safe state – defined by we have enough money to satisfy at least one person. Each of these “loans” results in a safe state. If we have exhausted our reserve, one person is working and will give us money greater than or equal to our previous “loan”, thus putting us in a safe state again. Since we always have the ability to make one additional move the system can never deadlock. Now, there is no guarentee that the system won’t livelock. If the process we hope to request something never does, no work will be done – but not due to deadlock. This analogy expands to higher orders of magnitude but requires that either a process can do its work entirely or there exists a process whose combination of resources can be satisfied, which makes the algorithm a little more tricky (an additional for loop) but nothing too bad. There are a fair bit of downsides to this</p>
<ul>
<li><p>The program first needs to know how much of each resource a process needs. A lot of times that is impossible or the process requests the wrong amount because the programmer didn’t forsee it.</p></li>
<li><p>The system could livelock.</p></li>
<li><p>We know in most systems that resources are generally not homogenous. Of course there are things like pipes and sockets but for the most part there is only 1 of a particular file. This could mean that the runtime of the algorithm could be slow for systems with millions of resources.</p></li>
<li><p>Also, this can’t keep track of resources that come and go. A process may delete a resource as a side effect or create a resource. The algorithm assumes a static allocation and that each process performs a non-destructive operation.</p></li>
</ul>
<h2 id="dining-philosophers">Dining Philosophers</h2>
<p>The Dining Philosophers problem is a classic synchronization problem. Imagine I invite <span class="math inline"><em>n</em></span> (let’s say 5) philosophers to a meal. We will sit them at a table with 5 chopsticks, one between each philosopher. A philosopher alternates between wanting to eat or think. To eat the philosopher must pick up the two chopsticks either side of their position. The original problem required each philosopher to have two forks, but one can eat with a single fork so we rule this out. However these chopsticks are shared with his neighbor.</p>
<p>[10]<span>r</span><span>.3</span></p>
<p><img src="deadlock/images/5DiningPhilosophers.png" alt="image" /></p>
<p>Is it possible to design an efficient solution such that all philosophers get to eat? Or, will some philosophers starve, never obtaining a second chopstick? Or will all of them deadlock? For example, imagine each guest picks up the chopstick on their left and then waits for the chopstick on their right to be free. Oops - our philosophers have deadlocked! Each of the philosophers are essentially the same, meaning that each philosopher has the same instruction set based on the other philosopher ie you can’t tell every even philosopher to do one thing and every odd philosopher to do another thing.</p>
<h3 id="failed-solutions">Failed Solutions</h3>
<p>[language=C] void* philosopher(void* forks)<span> info phil_info = forks; pthread_mutex_t* left_fork = phil_info-&gt;left_fork; pthread_mutex_t* right_fork = phil_info-&gt;right_fork; while(phil_info-&gt;simulation)<span> pthread_mutex_lock(left_fork); pthread_mutex_lock(right_fork); eat(left_fork, right_fork); pthread_mutex_unlock(left_fork); pthread_mutex_unlock(right_fork); </span> </span></p>
<p>This looks good but. What if everyone picks up their left fork and is waiting on their right fork? We have deadlocked the program. It is important to note that deadlock doesn’t happen all the time and the probability that this solution deadlocks goes down as the number of philosophers goes up. What is really important to note is that eventually that this solution will deadlock, letting threads starve which is bad. So now you are thinking about breaking one of the coffman conditions. Let’s break Hold and Wait!</p>
<p>[language=C] void* philosopher(void* forks)<span> info phil_info = forks; pthread_mutex_t* left_fork = phil_info-&gt;left_fork; pthread_mutex_t* right_fork = phil_info-&gt;right_fork; while(phil_info-&gt;simulation)<span> pthread_mutex_lock(left_fork); pthread_mutex_lock(right_fork); eat(left_fork, right_fork); pthread_mutex_unlock(left_fork); pthread_mutex_unlock(right_fork); </span> </span></p>
<p>Now our philosopher picks up the left fork and tries to grab the right. If it’s available, they eat. If it’s not available, they put the left fork down and try again. No deadlock! But, there is a problem. What if all the philosophers pick up their left at the same time, try to grab their right, put their left down, pick up their left, try to grab their right…. We have now livelocked our solution! Our poor philosopher are still starving, so let’s give them some proper solutions.</p>
<h2 id="viable-solutions">Viable Solutions</h2>
<p>The naive arbitrator solution is have one arbitrator (a mutex for example). Have each of the philosopher ask the arbitrator for permission to eat (i.e. trylock the mutex). This solution allows one philosopher to eat at a time. When they are done, another philosopher can ask for permission to eat. This prevents deadlock because there is no circular wait! No philosopher has to wait on any other philosopher. The advanced arbitrator solution is to implement a class that determines if the philosopher’s forks are in the arbitrator’s possession. If they are, they give them to the philosopher, let him eat, and take the forks back. This has the added bonus of being able to have multiple philosopher eat at the same time.</p>
<p>There are a lot of problems with these solutions. One is that they are slow and have a single point of failure or the arbitrator. Assuming that all the philosophers are good-willed, the arbitrator needs to be fair and be able to determine if a transaction would cause deadlock in the multi-arbitrator case. Further more in practical systems, the arbitrator tends to give forks to the same processes because of scheduling or pseudorandomness. Another important thing to note is that this prevents deadlock for the entire system. But in our model of dining philosophers, the philosopher has to release the lock themselves. Then you can consider the case of the malicious philosopher (let’s say Decartes because of his Evil Demons) could hold on to the arbitrator forever. He would make forward progress and the system would make forward progress but there is no way of ensuring that each process makes forward progress without assuming something about the processes or having true preemption – meaning that a higher authority (let’s say Steve Jobs) tells them to stop eating forcibly.</p>
<h3 id="leaving-the-table-stallings-solution">Leaving the Table (Stallings’ Solution)</h3>
<p>Why does the first solution deadlock? Well there are <span class="math inline"><em>n</em></span> philosophers and <span class="math inline"><em>n</em></span> chopsticks. What if there is only 1 philsopher at the table? Can we deadlock? No. How about 2 philsophers? 3? … You can see where this is going. Stallings’ <span class="citation"></span> solutions says to remove philosophers from the table until deadlock is not possible – think about what the magic number of philosophers at the table. The way to do this in actual system is through semaphores and letting a certain number of philosopher through. This has the benefit that multiple philosophers can be eating.</p>
<p>In the case that the philosophers aren’t evil, this solution requires a lot of time-consuming context switching. There is also no reliable way to know the number of resources before hand. In the dining philosophers case, this is solved because everything is known but trying to specify and operating system where you don’t know which file is going to get opened by what process leads you with a faulty solution. And again since semaphores are system constructs, they obey system timing clocks which means that the same processes tend to get added back into the queue again. Now if a philosopher becomes evil, then the problem becomes that there is no preemption. A philosopher can eat for as long as they want and the system will continue to function but that means the fairness of this solution can be low in the worst case. This works best with timeouts (or forced context switches) in order to ensure bounded wait times.</p>
<p>Stallings’ Solution Doesn’t Deadlock.</p>
<p>Let’s number the philosophers <span class="math inline">{<em>p</em><sub>0</sub>, <em>p</em><sub>1</sub>, ..,<em>p</em><sub><em>n</em> − 1</sub>}</span> and the resources <span class="math inline">{<em>r</em><sub>0</sub>, <em>r</em><sub>1</sub>, ..,<em>r</em><sub><em>n</em> − 1</sub>}</span>. A philosopher <span class="math inline"><em>p</em><sub><em>i</em></sub></span> needs resource <span class="math inline"><em>r</em><sub><em>i</em> − 1mod<em>n</em></sub></span> and <span class="math inline"><em>r</em><sub><em>i</em> + 1mod<em>n</em></sub></span>. Without loss of generality, let us take <span class="math inline"><em>p</em><sub><em>i</em></sub></span> out of the picture. Each resource had exactly two philosophers that could use it. Now resources <span class="math inline"><em>r</em><sub><em>i</em> − 1mod<em>n</em></sub></span> and <span class="math inline"><em>r</em><sub><em>i</em> + 1mod<em>n</em></sub></span> only have on philosopher waiting on it. Even if hold and wait, no preemption, and mutual exclusion or present, the resources can never enter a state where one philosopher requests them and they are held by another philosopher because only one philosopher can request them. Since there is no way to generate a cycle otherwise, circular wait cannot hold. Since circular wait cannot hold, deadlock cannot happen.</p>
<h3 id="partial-ordering-dijkstras-solution">Partial Ordering (Dijkstra’s Solution)</h3>
<p>This is Dijkstra’s solution <span class="citation"></span>. He was the one to propose this problem on an exam. Why does the first solution deadlock? Dijkstra thought that the last philosopher who picks up his left fork (causing the solution to deadlock) should pick up his right. He accomplishes it by number the forks <span class="math inline">1..<em>n</em></span>, and tells each of the philosopher to pick up his lower number fork. Let’s run through the deadlock condition again. Everyone tries to pick up their lower number fork first. Philosopher <span class="math inline">1</span> gets fork <span class="math inline">1</span>, Philosopher <span class="math inline">2</span> gets fork <span class="math inline">2</span>, and so on until we get to Philosopher <span class="math inline"><em>n</em></span>. They have to choose between fork <span class="math inline">1</span> and <span class="math inline"><em>n</em></span>. fork <span class="math inline">1</span> is already held up by philosopher <span class="math inline">1</span>, so they can’t pick up that fork, meaning he won’t pick up fork <span class="math inline"><em>n</em></span>. We have broken circular wait! Meaning deadlock isn’t possible.</p>
<p>The problems to this is that an entity either needs to know the finite set of resources or be able to produce a consistent partial order suck that circular wait cannot happen. This also implies that there needs to be some entity, either the operating system or another process, deciding on the number and all of the philosophers need to agree on the number as new resources come in. As we have also see with previous solutions, this relies on context switching so this prioritizes philosophers that have already eaten but can be made more fair by introducing random sleeps and waits.</p>
<h3 id="advanced-solutions">Advanced Solutions</h3>
<p>There are many more advanced solutions a non-exhaustive list includes</p>
<ul>
<li><p>Clean/Dirty Forks (Chandra/Misra Solution)</p></li>
<li><p>Actor Model (other Message passing models)</p></li>
</ul>
<h2 id="topics">Topics</h2>
<ul>
<li><p>Coffman Conditions</p></li>
<li><p>Resource Allocation Graphs</p></li>
<li><p>Dining Philosophers</p></li>
<li><p>Failed DP Solutions</p></li>
<li><p>Livelocking DP Solutions</p></li>
<li><p>Working DP Solutions: Benefits/Drawbacks</p></li>
</ul>
<h2 id="questions">Questions</h2>
<ul>
<li><p>What are the Coffman Conditions?</p></li>
<li><p>What do each of the Coffman conditions mean? (e.g. can you provide a definition of each one)</p></li>
<li><p>Give a real life example of breaking each Coffman condition in turn. A situation to consider: Painters, Paint, Paintbrushes etc. How would you assure that work would get done?</p></li>
<li><p>Be able to identify when Dining Philosophers code causes a deadlock (or not). For example, if you saw the following code snippet which Coffman condition is not satisfied?</p>
<p>[language=C] // Get both locks or none pthread_mutex_lock(a); if(pthread_mutex_trylock( b )) <span> /* failure */ pthread_mutex_unlock( a ); </span></p></li>
<li><p>The following calls are made</p>
<p>[language=c] // Thread 1 pthread_mutex_lock(m1) // success pthread_mutex_lock(m2) // blocks</p>
<p>// Thread 2 pthread_mutex_lock(m2) // success pthread_mutex_lock(m1) // blocks</p>
<p>What happens and why? What happens if a third thread calls <code>pthread_mutex_lock(m1)</code> ?</p></li>
<li><p>How many processes are blocked? As usual assume that a process is able to complete if it is able to acquire all of the resources listed below.</p>
<ul>
<li><p>P1 acquires R1</p></li>
<li><p>P2 acquires R2</p></li>
<li><p>P1 acquires R3</p></li>
<li><p>P2 waits for R3</p></li>
<li><p>P3 acquires R5</p></li>
<li><p>P1 waits for R4</p></li>
<li><p>P3 waits for R1</p></li>
<li><p>P4 waits for R5</p></li>
<li><p>P5 waits for R1</p></li>
</ul></li>
</ul>
<p>(Draw out the resource graph!)</p>
</body>
</html>
deadlock/deadlock.tex
+15 -10
View File
@@ -6,7 +6,7 @@
\\But if you try sometime you find
\\You get what you need}{The philosphers Jagger \& Richards}
Deadlock is defined as when the system cannot make and forward progress. In a lot of systems, Deadlock is just avoided by ignore the entire concept \cite[P.237]{silberschatz2006operating}. Have you heard about turn it on and off again? That is partly because of this. For products where the stakes are low when you deadlock (User Operating Systems, Phones), it may be more efficient not not keep track of all of the allocations in order to keep deadlock from happening. But in the cases where "failure is not an option" - Apollo 13, you need a system that tracks deadlock or better yet prevents it entirely. Take the Apollo 13 module. It may have not failed because of deadlock, but probably wouldn't be good to restart the system on liftoff.
\gls{Deadlock} is defined as when the system cannot make and forward progress. In a lot of systems, Deadlock is just avoided by ignore the entire concept \cite[P.237]{silberschatz2006operating}. Have you heard about turn it on and off again? That is partly because of this. For products where the stakes are low when you deadlock (User Operating Systems, Phones), it may be more efficient not not keep track of all of the allocations in order to keep deadlock from happening. But in the cases where "failure is not an option" - Apollo 13, you need a system that tracks deadlock or better yet prevents it entirely. Take the Apollo 13 module. It may have not failed because of deadlock, but probably wouldn't be good to restart the system on liftoff.
Mission critical operating systems need this guarentee formally because playing the odds with people's lives isn't a good idea. Okay so how do we do this? We model the problem. Even though it is a common statistical phrase that all models are wrong, the more accurate the model is to the system that we are working with the better chance that it'll work better.
@@ -19,7 +19,7 @@ Mission critical operating systems need this guarentee formally because playing
\caption{Resource allocation graph}
\end{wrapfigure}
One such way is modeling the system with a resource allocationg graph. A resource allocation graph tracks which resource is held by which process and which process is waiting for a resource of a particular type. It is very powerful and simple tool to illustrate how interacting processes can deadlock. If a process is \emph{using} a resource, an arrow is drawn from the resource node to the process node. If a process is \emph{requesting} a resource, an arrow is drawn from the process node to the resource node.If there is a cycle in the Resource Allocation Graph and each resource in the cycle provides only one instance, then the processes will deadlock. For example, if process 1 holds resource A, process 2 holds resource B and process 1 is waiting for B and process 2 is waiting for A, then process 1 and 2 process will be deadlocked.
One such way is modeling the system with a resource allocation graph (\gls{RAG}). A resource allocation graph tracks which resource is held by which process and which process is waiting for a resource of a particular type. It is very powerful and simple tool to illustrate how interacting processes can deadlock. If a process is \emph{using} a resource, an arrow is drawn from the resource node to the process node. If a process is \emph{requesting} a resource, an arrow is drawn from the process node to the resource node.If there is a cycle in the Resource Allocation Graph and each resource in the cycle provides only one instance, then the processes will deadlock. For example, if process 1 holds resource A, process 2 holds resource B and process 1 is waiting for B and process 2 is waiting for A, then process 1 and 2 process will be deadlocked.
\begin{wrapfigure}[12]{r}{.35\textwidth}
\begin{center}
@@ -32,18 +32,18 @@ Consider the following resource allocation graph. Assume that the processes ask
\section{Coffman conditions}
There are four \emph{necessary} and \emph{sufficient} conditions for deadlock -- meaning if these conditions hold then there is a non-zero probability that the system will deadlock at any given iteration. These are known as the Coffman conditions \cite{coffman1971system}.
There are four \emph{necessary} and \emph{sufficient} conditions for deadlock -- meaning if these conditions hold then there is a non-zero probability that the system will deadlock at any given iteration. These are known as the \gls{Coffman Conditions} \cite{coffman1971system}.
\begin{itemize}
\tightlist
\item
Mutual Exclusion: no two processes can obtain a resource at the same time.
\gls{Mutual Exclusion}: no two processes can obtain a resource at the same time.
\item
Circular Wait: there exists a cycle in the Resource Allocation Graph, or there exists a set of processes \{P1,P2,\ldots{}\} such that P1 is waiting for resources held by P2, which is waiting for P3,\ldots{}, which is waiting for P1.
\gls{Circular Wait}: there exists a cycle in the Resource Allocation Graph, or there exists a set of processes \{P1,P2,\ldots{}\} such that P1 is waiting for resources held by P2, which is waiting for P3,\ldots{}, which is waiting for P1.
\item
Hold and Wait: a process once obtaining a resource does not let go.
\gls{Hold and Wait}: a process once obtaining a resource does not let go.
\item
No pre-emption: nothing can force the process to give up a resource.
\gls{No pre-emption}: nothing can force the process to give up a resource.
\end{itemize}
\begin{proof} Deadlock can happen if and only if the four coffman conditions are satisified.
@@ -76,11 +76,11 @@ The last condition to address is circular wait. Circular wait means that there e
If you break any of them, you cannot have deadlock! Consider the scenario where two students need to write both pen and paper and there is only one of each. Breaking mutual exclusion means that the students share the pen and paper. Breaking circular wait could be that the students agree to grab the pen then the paper. As a proof by contradiction, say that deadlock occurs under the rule and the conditions. Without loss of generality, that means a student would have to be waiting on a pen while holding the paper and the other waiting on a pen and holding the paper. We have contradicted ourselves because one student grabbed the paper without grabbing the pen, so deadlock must not be able to occur. Breaking hold and wait could be that the students try to get the pen and then the paper and if a student fails to grab the paper then they release the pen. This introduces a new problem called \textit{livelock} which will be discussed latter. Breaking preemption means that if the two students are in deadlock the teacher can come in and break up the deadlock by giving one of the students one the held on items or tell both students to put the items down.
Livelock relates to deadlock but it is not exactly deadlock. Consider the breaking hold and wait solution as above. Though deadlock is avoided, if we pick up the same device (a pen or the paper) again and again in the exact same pattern, neither of us will get any writing done. More generally, livelock happens when the process looks like it is executing but no meaningful work is done. Livelock is generally harder to detect because the processes generally look like they are working to the outside operating system wheras in deadlock the operating system generally knows when two processes are waiting on a system wide resource. Another problem is that there are nessisary conditions for livelock (i.e. deadlock does not occur) but not sufficient conditions -- meaning there is no set of rules where livelock has to occur. You must formally prove in a system by what is known as an invariant. One has to enumerate each of the steps of a system and if each of the steps eventually (after some finite number of steps) leads to forward progress, the system is not livelocked. There are even better systems that prove bounded waits; a system can only be livelocked for at most $n$ cycles which may be important for something like stock exchanges.
\gls{livelock} relates to deadlock but it is not exactly deadlock. Consider the breaking hold and wait solution as above. Though deadlock is avoided, if we pick up the same device (a pen or the paper) again and again in the exact same pattern, neither of us will get any writing done. More generally, livelock happens when the process looks like it is executing but no meaningful work is done. Livelock is generally harder to detect because the processes generally look like they are working to the outside operating system wheras in deadlock the operating system generally knows when two processes are waiting on a system wide resource. Another problem is that there are nessisary conditions for livelock (i.e. deadlock does not occur) but not sufficient conditions -- meaning there is no set of rules where livelock has to occur. You must formally prove in a system by what is known as an invariant. One has to enumerate each of the steps of a system and if each of the steps eventually (after some finite number of steps) leads to forward progress, the system is not livelocked. There are even better systems that prove bounded waits; a system can only be livelocked for at most $n$ cycles which may be important for something like stock exchanges.
\section{Approaches to solving deadlock}
Ignoring deadlock is the most obvious approach that started the chapter out detailing. Quite humorously, the name for this approach is called the ostrich algorithm. Though there is no apparent source, the idea for the algorithm comes from the concept of an ostrich sticking its head in the sand. When the operating system detects deadlock, it does nothing out of the ordinary and hopes that the deadlock goes away. Now this is a slight misnomer because the operating system doesn't do anything \textit{abnormal} -- it is not like an operating system deadlocks every few minutes because it runs ~100 processes all requesting shared libraries. An operating system still preempts processes when stopping them for context switches. The operating system has the ability to interrupt any system call, potentially breaking a deadlock scenario. The OS also makes some files read-only thus making the resource shareable. What the algorithm refers to is that if there is an adversary that specifically crafts a program -- or equivalently a user who poorly writes a program -- that deadlock could not be caught by the operating system. For everyday life, this tends to be fine. When it is not we can turn to the following method.
Ignoring deadlock is the most obvious approach that started the chapter out detailing. Quite humorously, the name for this approach is called the \gls{ostrich algorithm}. Though there is no apparent source, the idea for the algorithm comes from the concept of an ostrich sticking its head in the sand. When the operating system detects deadlock, it does nothing out of the ordinary and hopes that the deadlock goes away. Now this is a slight misnomer because the operating system doesn't do anything \textit{abnormal} -- it is not like an operating system deadlocks every few minutes because it runs ~100 processes all requesting shared libraries. An operating system still preempts processes when stopping them for context switches. The operating system has the ability to interrupt any system call, potentially breaking a deadlock scenario. The OS also makes some files read-only thus making the resource shareable. What the algorithm refers to is that if there is an adversary that specifically crafts a program -- or equivalently a user who poorly writes a program -- that deadlock could not be caught by the operating system. For everyday life, this tends to be fine. When it is not we can turn to the following method.
Deadlock detection allows the system to enter a deadlocked state. After entering, the system uses the information that it has to break deadlock. As an example, consider multiple processes accessing files. The operating system is able to keep track of all of the files/resources through file descriptors at some level either abstracted through an API or directly. If the operating system detects a directed cycle in the operating system file descriptor table it may break one process' hold through scheduling for example and let the system proceed. Why this is a popular choice in this realm is that there is no way of knowing which resources a program will select without running the program. This is an extension of Rice's theorem \cite{rice} that says that we cannot know any semantic feature without running the program (semantic meaning like what files it tries to open). So theoretically, it is sound. The problem then gets introduced that we could reach a livelock scenario if we preempt a set of resources again and again. The way around this is mostly probabilistic. The operating system chooses a random resource to break hold and wait. Now even though a user can craft a program where breaking hold and wait on each resource will result in a livelock, this doesn't happen as often on machines that run programs in practice or the livelock that does happen happens for a couple of cycles. These kind of systems are good for products that need to maintain a non-deadlocked state but can tolerate a small chance of livelock for a short period of time. The following proof \textbf{is not required for our 241 related puproses but is included for concreteness}.
@@ -200,7 +200,12 @@ Why does the first solution deadlock? Well there are $n$ philosophers and $n$ ch
In the case that the philosophers aren't evil, this solution requires a lot of time-consuming context switching. There is also no reliable way to know the number of resources before hand. In the dining philosophers case, this is solved because everything is known but trying to specify and operating system where you don't know which file is going to get opened by what process leads you with a faulty solution. And again since semaphores are system constructs, they obey system timing clocks which means that the same processes tend to get added back into the queue again. Now if a philosopher becomes evil, then the problem becomes that there is no preemption. A philosopher can eat for as long as they want and the system will continue to function but that means the fairness of this solution can be low in the worst case. This works best with timeouts (or forced context switches) in order to ensure bounded wait times.
\todo{Prove stallings's doesn't deadlock}
\begin{proof} Stallings' Solution Doesn't Deadlock.
Let's number the philosophers $\{p_0, p_1, .., p_{n-1}\}$ and the resources $\{r_0, r_1, .., r_{n-1}\}$. A philosopher $p_i$ needs resource $r_{i-1 \mod n}$ and $r_{i + 1 \mod n}$. Without loss of generality, let us take $p_i$ out of the picture. Each resource had exactly two philosophers that could use it. Now resources $r_{i-1 \mod n}$ and $r_{i + 1 \mod n}$ only have on philosopher waiting on it. Even if hold and wait, no preemption, and mutual exclusion or present, the resources can never enter a state where one philosopher requests them and they are held by another philosopher because only one philosopher can request them. Since there is no way to generate a cycle otherwise, circular wait cannot hold. Since circular wait cannot hold, deadlock cannot happen.
\end{proof}
\subsection{Partial Ordering (Dijkstra's Solution)}
glossary.tex
+43 -1
View File
@@ -10,8 +10,50 @@
description={TODO}
}
% Deadlock
\newglossaryentry{livelock}
{
name=livelock,
name=Livelock,
description={TODO}
}
\newglossaryentry{Deadlock}{
name=Deadlock,
description={When a system cannot progress}
}
\newglossaryentry{RAG}{
name=RAG,
description={A tool for helping identify deadlock if a cycle is apprent}
}
\newglossaryentry{Coffman Conditions}{
name=Coffman Conditions,
description={Four necissary and sufficient conditions for deadlock}
}
\newglossaryentry{Mutual Exclusion}{
name=a,
description={a}
}
\newglossaryentry{Circular Wait}{
name=a,
description={a}
}
\newglossaryentry{Hold and Wait}{
name=a,
description={a}
}
\newglossaryentry{No pre-emption}{
name=a,
description={a}
}
\newglossaryentry{ostrich algorithm}{
name=a,
description={a}
}
introc/introc.bib
+24
View File
@@ -56,3 +56,27 @@
year={1988},
publisher={Prentice Hall}
}
@techreport{ISON1124,
type = {Standard},
key = {ISO 1124:2005},
month = mar,
year = {2005},
title = {{ISO C Standard}},
address = {Geneva, CH},
url={http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1124.pdf},
institution = {International Organization for Standardization}
}
@ARTICLE{4610935,
author={},
journal={IEEE Std 754-2008},
title={IEEE Standard for Floating-Point Arithmetic},
year={2008},
volume={},
number={},
pages={1-70},
keywords={IEEE standards;floating point arithmetic;programming;IEEE standard;arithmetic formats;computer programming;decimal floating-point arithmetic;754-2008;NaN;arithmetic;binary;computer;decimal;exponent;floating-point;format;interchange;number;rounding;significand;subnormal},
doi={10.1109/IEEESTD.2008.4610935},
ISSN={},
month={Aug},}
introc/introc.tex
+424 -300
View File
@@ -10,7 +10,7 @@ C was developed by Dennis Ritchie and Ken Thompson at Bell Labs back in 1973 \ci
The first "real" standardization is with Brian Kerninghan and Dennis Ritchies book \cite{kernighan1988c}. It is still widely regarded today as the only \gls{portable} set of C instructions. The K\&R book is known as the de-facto standard for learning C. There were different standards of C from ANSI to ISO after the Unix guides. The one that we will be mainly focusing on is the \gls{POSIX} C library. Now to get the elephant out of the room, the Linux kernel is not entirely POSIX compliant. Mostly, it is because they didn't want to pay the fee for compliance
\todo
\todo{Further the history until today}
\section{Features}
@@ -160,20 +160,194 @@ do {
} while (i > 10) /* Only executed once */
\end{code}
\item \keyword{enum}
\item \keyword{extern}
\item \keyword{float}
\item \keyword{for}
\item \keyword{goto}
\item \keyword{if else else-if} are control flow keywords. There are a few ways to use these
\item \keyword{inline}
\item \keyword{register}
\item \keyword{restrict}
\item \keyword{enum} is to declare an enumeration. An enumeration is a type that can take on many, finite values. If you have an enum and don't specify any numerics, the c compiler when generate a unique number for that enum (within the context of the current enum) and use that for comparisons. To declare an instance of an enum, you must say \keyword{enum <type> varname}. The added benefit to this is that C can type check these expressions to make sure that you are only comparing alike types.
\begin{code}[language=C]
enum day{ monday, tuesday, wednesday,
thursday, friday, saturday, sunday};
void process_day(enum day foo) {
switch(day) {
case monday:
printf("Go home!\n"); break;
// ...
}
}
\end{code}
It is completely possible to assign enum values to either be different or the same. Just don't rely on the compiler for consistent numbering. If you are going to use this abstraction, try not to break it.
\begin{code}[language=C]
enum day{
monday = 0,
tuesday = 0,
wednesday = 0,
thursday = 1,
friday = 10,
saturday = 10,
sunday = 0};
void process_day(enum day foo) {
switch(day) {
case monday:
printf("Go home!\n"); break;
// ...
}
}
\end{code}
\item \keyword{extern} is a special keyword that tells the compiler that the variable may be defined in another object file or a library, so the compiler doesn't throw an error when either the variable is not defined or if the variable is defined twice because the first file will really be referencing the variable in the other file.
\begin{code}[language=C]
// file1.c
extern int panic;
void foo() {
if (panic) {
printf("NONONONONO");
} else {
printf("This is fine");
}
}
//file2.c
int panic = 1;
\end{code}
\item \keyword{for} is a keyword that allows you to iterate with an initialization condition, a loop invariant, and an update condition. This is meant to be a replacement for the while loop
\begin{code}[language=C]
for (initialization; check; update) {
//...
}
// Typically
int i;
for (i = 0; i < 10; i++) {
//...
}
\end{code}
One thing to note is that as of the C89 standard, you cannot declare variables inside the \keyword{for} loop. This is because there was a disagreement in the standard for how the scoping rules of a variable defined in the loop would work. It has since been resolved with more recent standards, so people can use the for loop that they know and love today
\begin{code}[language=C]
for(int i = 0; i < 10; ++i) {
}
\end{code}
The order of evaluation for a \keyword{for} loop is as follows
\begin{enumerate}
\item Perform the initialization condition.
\item Check the invariant. If false, terminate the loop and execute the next statement. If true, continue to the body of the loop.
\item Perform the body of the loop.
\item Perform the update condition.
\item Jump to (2).
\end{enumerate}
\item \keyword{goto} is a keyword that allows you to do conditional jumps. Do not use \keyword{goto} in your programs. The reason being is that it makes your code infinitely more hard to understand when strung together with multiple chains. It is fine to use in some contexts though. The keyword is usually used in kernel contexts when adding another stack frame for cleanup isn't a good idea. The canonical example of kernel cleanup is as below.
\begin{code}[language=C]
void setup(void) {
Doe *deer;
Ray *drop;
Mi *myself;
if (!setupdoe(deer)) {
goto finish;
}
if (!setupray(drop)) {
goto cleanupdoe;
}
if (!setupmi(myself)) {
goto cleanupray;
}
perform_action(deer, drop, myself);
cleanupray:
cleanup(drop);
cleanupdoe:
cleanup(deer);
finish:
return;
}
\end{code}
\item \keyword{if else else-if} are control flow keywords. There are a few ways to use these (1) A bare if (2) An if with an else (3) an if with an else-if (4) an if with an else if and else. The statements are always executed from the if to the else. If any of the intermediate conditions are true, the if block performs that action and goes to the end of that block.
\begin{code}[language=C]
// (1)
if (connect(...))
return -1;
// (2)
if (connect(...)) {
exit(-1);
} else {
printf("Connected!");
}
// (3)
if (connect(...)) {
exit(-1);
} else if (bind(..)) {
exit(-2);
}
// (1)
if (connect(...)) {
exit(-1);
} else if (bind(..)) {
exit(-2);
} else {
printf("Successfully bound!");
}
\end{code}
\item \keyword{inline} is a compiler keyword that tells the compiler it's okay not to create a new function in the assembly. Instead, the compile is hinted at substituting the function body directly into the calling function. This is not always recommended explicitly as the compiler is usually smart enough to know when to \keyword{inline} a function for you.
\begin{code}[language=C]
inline int max(int a, int b) {
return a < b ? a : b;
}
int main() {
printf("Max %d", max(a, b));
// printf("Max %d", a < b ? a : b);
}
\end{code}
\item \keyword{restrict} is a keyword that tells the compiler that this particular memory region shouldn't overlap with all other memory regions. The use case for this is to tell users of the program that it is undefined behavior if the memory regions overlap.
\begin{code}[language=C]
memcpy(void * restrict dest, const void* restrict src, size_t bytes);
void add_array(int *a, int * restrict c) {
*a += *c;
}
int *a = malloc(3*sizeof(*a));
*a = 1; *a = 2; *a = 3;
add_array(a + 1, a) // Well defined
add_array(a, a) // Undefined
\end{code}
\item \keyword{return} is a control flow operator that exits the current function. If the function is \keyword{void} then it simply exits the functions. Otherwise another parameter follows as the return value.
\begin{code}[language=C]
void process() {
if (connect(...)) {
return -1;
} else if (bind(...)) {
return -2
}
return 0;
}
\end{code}
\item \keyword{signed} is a modifier which is rarely used, but it forces an type to be signed instead of unsigned. The reasont that this is so rarely used is because types are signed by default and need to have the \keyword{unsigned} modifier to make them unsigned but it may be useful in cases where you want the compiler to default a signed type like.
\begin{code}[language=C]
@@ -193,7 +367,7 @@ printf("%zu", 1);
\end{code}
Which then the compiler is allowed to operate on further. A note that you must have a complete definition of the type at compile time or else you may get an odd error. Consider the following
\\
\begin{code}[language=C]
// file.c
struct person;
@@ -207,9 +381,7 @@ struct person {
}
\end{code}
This code will not compile because sizeof is not able to compile \keyword{file.c} without knowing the full declaration of the \texttt{person} struct. That is typically why we either put the full declaration in a header file or we abstract the creation and the interaction away so that users cannot access the internals of our struct.
\keyword{sizeof(ary)}: \keyword{ary} is an array. Returns the number of bytes required for the entire array (5 chars + zero byte = 6 bytes) \keyword{sizeof(ptr)}: Same as sizeof(char *). Returns the number bytes required for a pointer (e.g.~4 or 8 for a 32 bit or 64-bit machine) \keyword{sizeof} is a special operator. Really it's something the compiler substitutes in before compiling the program because the size of all types is known at compile time. When you have \keyword{sizeof(char*)} that takes the size of a pointer on your machine (8 bytes for a 64-bit machine and 4 for a 32 bit and so on). When you try \keyword{sizeof(char{[}{]})}, the compiler looks at that and substitutes the number of bytes that the \textbf{entire} array contains because the total size of the array is known at compile time.
This code will not compile because sizeof is not able to compile \keyword{file.c} without knowing the full declaration of the \texttt{person} struct. That is typically why we either put the full declaration in a header file or we abstract the creation and the interaction away so that users cannot access the internals of our struct. Also, if the compiler knows the full length of an array object, it will use that in the expression instead of decaying it to a pointer.
\begin{code}[language=C]
char str1[] = "will be 11";
@@ -219,8 +391,45 @@ sizeof(str2) //8 because it is a pointer
\end{code}
Be careful, using sizeof for the length of a string!
\item \keyword{static}
\item \keyword{struct}
\item \keyword{static} is a type specifier with three meanings.
\begin{enuemrate}
\item When used with a global variable or function declaration it means that the scope of the variable or the function is only limited to the file.
\item When used with a function variable, that declares that the variable has static allocation -- meaning that the variable is allocated once at program start up not every time the program is run.
\end{enumerate}
\begin{code}[language=C]
static int i = 0;
static int _perform_calculation(void) {
// ...
}
char *print_time(void) {
static char buffer[200]; // Shared every time a function is called
// ...
}
\end{code}
\item \keyword{struct} is a keyword that allows you to pair multiple types together into a new structure. Structs are contiguous regions of memory that one can access specific elements of each memory as if they were separate variables.
\begin{code}[language=C]
struct hostname {
const char *port;
const char *name;
const char *resource;
}; // You need the semicolon at the end
// Assign each individually
struct hostname facebook;
facebook.port = "80";
facebook.name = "www.google.com";
facebook.resource = "/";
// You can use static initialization in later versions of c
struct hostname google = {"80", "www.google.com", "/"};
\end{code}
\item \keyword{switch case default} Switches are essentially glorified jump statements. Meaning that you take either a byte or an integer and the control flow of the program jumps to that location.
\\
\begin{code}[language=C]
@@ -255,8 +464,7 @@ typedef struct link link_t;
//With structs, include the keyword 'struct' as part of the original types
\end{code}
In this class, we regularly typedef functions. A typedef for a function
can be this for example
In this class, we regularly typedef functions. A typedef for a function can be this for example
\begin{code}[language=C]
typedef int (*comparator)(void*,void*);
@@ -269,8 +477,28 @@ comparator gt = greater_than;
This declares a function type comparator that accepts two \keyword{void*} params and returns an integer.
\item \keyword{union}
\item \keyword{unsigned}
\item \keyword{union} is a new type specifier. A union is one piece of memory that a bunch of variables occupy. It is used to maintain consistency while having the flexibility to switch between types without mainting functions to keep track of the bits. Consider an example where we have different pixel values.
\begin{code}[language=C]
union pixel {
struct values {
char red;
char blue;
char green;
char alpha;
} values;
uint32_t encoded;
}; // Ending semicolon needed
union pixel a;
// When modifying or reading
a.values.red;
a.values.blue = 0x0;
// When writing to a file
fprintf(picture, "%d", a.encoded);
\end{code}
\item \keyword{unsigned} is a type modifier that forces \keyword{unsigned} behavior in the variables they modify. Unsigned can only be on primitive int types (like \keyword{int} and \keyword{long}). There is a lot of behavior associated with unsigned arthmetic and whatnot, just know for the most part unless you need to do bit shifting you probably won't need it.
\item \keyword{void} is a two folded keyword. When used in terms of function or parameter definition then it means that it returns no value or accepts no parameter specifically. The following declares a function that accepts no parameters and returns nothing.
\begin{code}[language=C]
@@ -309,10 +537,10 @@ If you put the volatile keyword then it forces the compiler to keep the variable
\begin{enumerate}
\item \keyword{char} Represents exactly one byte of data. The number of bits in a byte might vary. \keyword{unsigned char} and \keyword{signed char} mean the exact same thing. This must be aligned on a boundary (meaning you cannot use bits in between two addresses). The rest of the types will assume 8 bits in a byte.
\item \keyword{short (short int)} must be at least two bytes. This is aligned on a two byte boundary, meaning that the address must be divisble by two.
\item \keyword{int} must be at least two bytes. Again aligned to a two byte boundary. \href{http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1124.pdf}{Page 34}. On most machines this will be 4 bytes.
\item \keyword{int} must be at least two bytes. Again aligned to a two byte boundary \cite[P. 34]{ISON1124}. On most machines this will be 4 bytes.
\item \keyword{long (long int)} must be at least four bytes, which are aligned to a four byte boundary. On some machines this can be 8 bytes.
\item \keyword{long long} must be at least eight bytes, aligned to an eight byte boundary.
\item \keyword{float} represents an IEEE-754 single percision floating point number tightly specified by \href{http://ieeexplore.ieee.org/document/4610935/}{IEEE}. This will be four bytes aligned to a four byte boundary on most machines.
\item \keyword{float} represents an IEEE-754 single percision floating point number tightly specified by IEEE \cite{4610935}. This will be four bytes aligned to a four byte boundary on most machines.
\item \keyword{double} represents an IEEE-754 double percision floating point number specified by the same standard, which is aligned to the nearest eight byte boundary.
\end{enumerate}
@@ -336,28 +564,20 @@ printf("Debug: The string and int are stored at: %p and %p\n", name, &score );
// We need "&" to get the address of the int variable
\end{code}
By default, for performance, \keyword{printf} does not actually write
anything out (by calling write) until its buffer is full or a newline is
printed.
By default, for performance, \keyword{printf} does not actually write anything out (by calling write) until its buffer is full or a newline is printed.
\subsection{How else can I print strings and single characters?}
Use \keyword{puts(\ name\ )} and \keyword{putchar(\ c\ )} where name is a
pointer to a C string and c is just a \keyword{char}
Use \keyword{puts(\ name\ )} and \keyword{putchar(\ c\ )} where name is a pointer to a C string and c is just a \keyword{char}
\subsection{How do I print to other file
streams?}\label{how-do-i-print-to-other-file-streams}
Use
\keyword{fprintf(\ \_file\_\ ,\ "Hello\ \%s,\ score:\ \%d",\ name,\ score);}
Where \_file\_ is either predefined `stdout' `stderr' or a FILE pointer
that was returned by \keyword{fopen} or \keyword{fdopen}
Use \keyword{fprintf(\ \_file\_\ ,\ "Hello\ \%s,\ score:\ \%d",\ name,\ score);} Where \_file\_ is either predefined `stdout' `stderr' or a FILE pointer that was returned by \keyword{fopen} or \keyword{fdopen}
\subsection{Can I use file descriptors?}\label{can-i-use-file-descriptors}
Yes! Just use \keyword{dprintf(int\ fd,\ char*\ format\_string,\ ...);}
Just remember the stream may be buffered, so you will need to assure
that the data is written to the file descriptor.
Yes! Just use \keyword{dprintf(int\ fd,\ char*\ format\_string,\ ...);} Just remember the stream may be buffered, so you will need to assure that the data is written to the file descriptor.
\subsection{How do I print data into a C string?}
@@ -368,23 +588,30 @@ char result[200];
int len = snprintf(result, sizeof(result), "%s:%d", name, score);
\end{code}
snprintf returns the number of characters written excluding the
terminating byte. In the above example, this would be a maximum of 199.
\keyword{snprintf} returns the number of characters written excluding the terminating byte. In the above example, this would be a maximum of 199.
The following code is vulnerable to buffer overflow. It assumes or
trusts that the input line will be no more than 10 characters, including
the terminating byte.
The following code is vulnerable to buffer overflow. It assumes or trusts that the input line will be no more than 10 characters, including the terminating byte.
\begin{code}[language=C]
char buf[10];
gets(buf); // Remember the array name means the first byte of the array
\end{code}
\subsection{What if I really really want printf to call write without a newline?}
Use \keyword{fflush(\ FILE*\ inp\ )}. The contents of the file will be written. If I wanted to write ``Hello World'' with no newline, I could write it like this.
\begin{code}[language=C]
int main(){
fprintf(stdout, "Hello World");
fflush(stdout);
return 0;
}
\end{code}
\subsection{Should I Use Gets?}
\keyword{gets} is deprecated in C99 standard and has been removed from
the latest C standard (C11). Programs should use \keyword{fgets} or
\keyword{getline} instead.
\keyword{gets} is deprecated in C99 standard and has been removed from the latest C standard (C11). Programs should use \keyword{fgets} or \keyword{getline} instead.
Where each has the following structure respectively:
@@ -394,17 +621,14 @@ char *fgets (char *str, int num, FILE *stream);
ssize_t getline(char **lineptr, size_t *n, FILE *stream);
\end{code}
Here's a simple, safe way to read a single line. Lines longer than 9
characters will be truncated:
Here's a simple, safe way to read a single line. Lines longer than 9 characters will be truncated:
\begin{code}[language=C]
char buffer[10];
char *result = fgets(buffer, sizeof(buffer), stdin);
\end{code}
The result is NULL if there was an error or the end of the file is
reached. Note, unlike \keyword{gets}, \keyword{fgets} copies the newline
into the buffer, which you may want to discard-
The result is NULL if there was an error or the end of the file is reached. Note, unlike \keyword{gets}, \keyword{fgets} copies the newline into the buffer, which you may want to discard-
\begin{code}[language=C]
if (!result) { return; /* no data - don't read the buffer contents */}
@@ -414,8 +638,7 @@ if (buffer[i] == '\n')
buffer[i] = '\0';
\end{code}
One of the advantages of \keyword{getline} is that will automatically
(re-) allocate a buffer on the heap of sufficient size.
One of the advantages of \keyword{getline} is that will automatically (re-) allocate a buffer on the heap of sufficient size.
\begin{code}[language=C]
// ssize_t getline(char **lineptr, size_t *n, FILE *stream);
@@ -443,11 +666,64 @@ free(buffer);
\subsection{string.h}
Why are \keyword{memcpy} and \keyword{memmove} both in \keyword{\textless{}string.h\textgreater{}}? Because strings are essentially raw memory with a null byte at the end of them! \keyword{void\ *memcpy(void\ *dest,\ const\ void\ *src,\ size\_t\ n)} moves \keyword{n} bytes starting at \keyword{src} to \keyword{dest}. \textbf{Be careful}, there is undefined behavior when the memory regions overlap. This is one of the classic works on my machine examples because many times valgrind won't be able to pick it up because it will look like it works on your machine. When the autograder hits, fail. Consider the safer version which is.
\href{https://linux.die.net/man/3/string}{More information about all of these functions}. Any behavior like passing \keyword{strlen(NULL)} is considered undefined behavior.
\keyword{void\ *memmove(void\ *dest,\ const\ void\ *src,\ size\_t\ n)}
does the same thing as above, but if the memory regions overlap then it
is guaranteed that all the bytes will get copied over correctly.
\begin{itemize}
\item \keyword{int\ strlen(const\ char\ *s)} returns the length of the string not including the null byte
\item \keyword{int\ strcmp(const\ char\ *s1,\ const\ char\ *s2)} returns an integer determining the lexicographic order of the strings. If s1 where to come before s2 in a dictionary, then a -1 is returned. If the two strings are equal, then 0. Else, 1.
\item \keyword{char\ *strcpy(char\ *dest,\ const\ char\ *src)} Copies the string at \keyword{src} to \keyword{dest}. \textbf{assumes dest has enough space for src}
\item \keyword{char\ *strcat(char\ *dest,\ const\ char\ *src)} Concatenates the string at \keyword{src} to the end of destination. \textbf{This function assumes that there is enough space for \keyword{src} at the end of destination including the NULL byte}
\item \keyword{char\ *strdup(const\ char\ *dest)} Returns a \keyword{malloc}'ed copy of the string.
\item \keyword{char\ *strchr(const\ char\ *haystack,\ int\ needle)} Returns a pointer to the first occurrence of \keyword{needle} in the \keyword{haystack}. If none found, \keyword{NULL} is returned.
\item \keyword{char\ *strstr(const\ char\ *haystack,\ const\ char\ *needle)} Same as above but this time a string!
\item \keyword{char *strtock(const char *str, const char *delims)}
A dangerous but useful function strtok takes a string and tokenizes it. Meaning that it will transform the strings into separate strings. This function has a lot of specs so please read the man pages a contrived example is below.
\begin{code}[language=C]
#include <stdio.h>
#include <string.h>
int main(){
char* upped = strdup("strtok,is,tricky,!!");
char* start = strtok(upped, ",");
do{
printf("%s\n", start);
}while((start = strtok(NULL, ",")));
return 0;
}
\end{code}
\textbf{Output}
\begin{code}[language=console]
strtok
is
tricky
!!
\end{code}
What happens when I change \keyword{upped} like this?
\begin{code}[language=C]
char* upped = strdup("strtok,is,tricky,,,!!");
\end{code}
\item \keyword{void\ *memcpy(void\ *dest,\ const\ void\ *src,\ size\_t\ n)} moves \keyword{n} bytes starting at \keyword{src} to \keyword{dest}. \textbf{Be careful}, there is undefined behavior when the memory regions overlap. This is one of the classic works on my machine examples because many times valgrind won't be able to pick it up because it will look like it works on your machine. When the autograder hits, fail. Consider the safer version below.
\item \keyword{void\ *memmove(void\ *dest,\ const\ void\ *src,\ size\_t\ n)} does the same thing as above, but if the memory regions overlap then it is guaranteed that all the bytes will get copied over correctly.
\end{itemize}
\keyword{memcpy} and \keyword{memmove} both in \keyword{\textless{}string.h\textgreater{}}? Because strings are essentially raw memory with a null byte at the end of them!
\subsection{stdlib.h}
@@ -459,12 +735,18 @@ int *space = malloc(sizeof(int) * 10);
\subsection{Conventions/Errno}
\todo{Conventions and errno}
\section{System Calls}
\subsection{What is a system call?}
\todo{System Call}
\subsection{The interplay with library functions}
\todo{System calls and library function}
\subsubsection{Does \keyword{printf} call write or does write call \keyword{printf}?}
\keyword{printf} calls \keyword{write}. \keyword{printf} includes an
@@ -478,14 +760,96 @@ buffer which suits our needs better at that point
\subsection{Structs}
\todo{What's a struct?}
In low-level terms, a struct is just a piece of contiguous memory, nothing more. Just like an array, a struct has enough space to keep all of its members. But unlike an array, it can store different types. Consider the contact struct declared above
\begin{code}[language=C]
struct contact {
char firstname[20];
char lastname[20];
unsigned int phone;
};
struct contact bhuvan;
\end{code}
\begin{aside}
\begin{code}[language=C]
/* a lot of times we will do the following typdef
so we can just write contact contact1 */
typedef struct contact contact;
contact bhuvan;
/* You can also declare the struct like this to get
it done in one statement */
typedef struct optional_name {
...
} contact;
\end{code}
If you compile the code without any optimizations and reordering, you can expect the addresses of each of the variables to look like this.
\begin{code}[language=C]
&bhuvan // 0x100
&bhuvan.firstname // 0x100 = 0x100+0x00
&bhuvan.lastname // 0x114 = 0x100+0x14
&bhuvan.phone // 0x128 = 0x100+0x28
\end{code}
Because all your compiler does is say `hey reserve this much space, and I will go and calculate the offsets of whatever variables you want to write to'. The offsets are where the variable starts at. The phone variables starts at the \keyword{0x128}th bytes and continues for sizeof(int) bytes, but not always. \textbf{Offsets don't determine where the variable ends though}. Consider the following hack that you see in a lot of kernel code.
\begin{code}[language=C]
typedef struct {
int length;
char c_str[0];
} string;
const char* to_convert = "bhuvan";
int length = strlen(to_convert);
// Let's convert to a c string
string* bhuvan_name;
bhuvan_name = malloc(sizeof(string) + length+1);
/*
Currently, our memory looks like this with junk in those black spaces
___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___
bhuvan_name = |___|___|___|___|___|___|___|___|___|___|___|
*/
bhuvan_name->length = length;
/*
This writes the following values to the first four bytes
The rest is still garbage
___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___
bhuvan_name = | 0 | 0 | 0 | 6 |___|___|___|___|___|___|___|
*/
strcpy(bhuvan_name->c_str, to_convert);
/*
Now our string is filled in correctly at the end of the struct
___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ____
bhuvan_name = | 0 | 0 | 0 | 6 | b | h | u | v | a | n | \0 |
‾
*/
strcmp(bhuvan_name->c_str, "bhuvan") == 0 //The strings are equal!
\end{code}
\end{aside}
\subsubsection{Struct packing}
Structs may require something called \href{http://www.catb.org/esr/structure-packing/}{padding} (tutorial).
\textbf{We do not expect you to pack structs in this course, just know that it
is there} This is because in the early days (and even now) when you have
to an address from memory you have to do it in 32bit or 64bit blocks.
This also meant that you could only request addresses that were
multiples of that. Meaning that
Structs may require something called \href{http://www.catb.org/esr/structure-packing/}{padding} (tutorial). \textbf{We do not expect you to pack structs in this course, just know that it is there} This is because in the early days (and even now) when you have to an address from memory you have to do it in 32bit or 64bit blocks. This also meant that you could only request addresses that were multiples of that. Meaning that
\begin{code}[language=C]
struct picture{
@@ -868,22 +1232,6 @@ The \keyword{&&} and \keyword{||} operator
\begin{comment}
\subsection{\texorpdfstring{What if I really really want \keyword{printf}
to call \keyword{write} without a
newline?}{What if I really really want printf to call write without a newline?}}\label{what-if-i-really-really-want-printf-to-call-write-without-a-newline}
Use \keyword{fflush(\ FILE*\ inp\ )}. The contents of the file will be
written. If I wanted to write ``Hello World'' with no newline, I could
write it like this.
\begin{code}[language=C]
int main(){
fprintf(stdout, "Hello World");
fflush(stdout);
return 0;
}
\end{code}
\subsection{\texorpdfstring{How is \keyword{perror}
helpful?}{How is perror helpful?}}\label{how-is-perror-helpful}
@@ -1098,25 +1446,6 @@ on to the next statement
\section{Other Gotchas}\label{other-gotchas}
\subsection{\texorpdfstring{Does \keyword{sizeof} do
anything?}{Does sizeof do anything?}}\label{does-sizeof-do-anything}
\begin{code}[language=C]
int a = 0;
size_t size = sizeof(a++);
printf("size: %lu, a: %d", size, a);
\end{code}
What does the code print out?
\begin{code}[language=C]
size: 4, a: 0
\end{code}
Because sizeof is not actually evaluated at runtime. The compiler
assigns the type of all expressions and discards the extra results of
the expression.
\section{Strings, Structs, and
Gotcha's}\label{strings-structs-and-gotchas}
@@ -1147,217 +1476,12 @@ that any attempt to modify the string will cause a segfault.
If one, however, \keyword{malloc}'s space, one can change that string to
be whatever they want.
\subsection{Memory Mismanagement}\label{memory-mismanagement}
One common gotcha is when you write the following
\begin{code}[language=C]
char* hello_string = malloc(14);
___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___
// hello_string ----> | g | a | r | b | a | g | e | g | a | r | b | a | g | e |
hello_string = "Hello Bhuvan!";
// (constant string in the text segment)
// hello_string ----> [ "H" , "e" , "l" , "l" , "o" , " " , "B" , "h" , "u" , "v" , "a" , "n" , "!" , "\0" ]
___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___
// memory_leak -----> | g | a | r | b | a | g | e | g | a | r | b | a | g | e |
hello_string[9] = 't'; //segfault!!
\end{code}
What did we do? We allocated space for 14 bytes, reassigned the pointer
and successfully segfaulted! Remember to keep track of what your
pointers are doing. What you probably wanted to do was use a
\keyword{string.h} function \keyword{strcpy}.
\begin{code}[language=C]
strcpy(hello_string, "Hello Bhuvan!");
\end{code}
\subsection{Remember the NULL byte!}\label{remember-the-null-byte}
Forgetting to NULL terminate a string is a big affect on the strings!
Bounds checking is important. The heart bleed bug mentioned earlier in
the wiki book is partially because of this.
\subsection{Where can I find an In-Depth and Assignment-Comprehensive
explanation of all of these
functions?}\label{where-can-i-find-an-in-depth-and-assignment-comprehensive-explanation-of-all-of-these-functions}
\href{https://linux.die.net/man/3/string}{Right Here!}
\subsection{\texorpdfstring{String Information/Comparison:
\keyword{strlen}
\keyword{strcmp}}{String Information/Comparison: strlen strcmp}}\label{string-informationcomparison-strlen-strcmp}
\keyword{int\ strlen(const\ char\ *s)} returns the length of the string
not including the null byte
\keyword{int\ strcmp(const\ char\ *s1,\ const\ char\ *s2)} returns an
integer determining the lexicographic order of the strings. If s1 where
to come before s2 in a dictionary, then a -1 is returned. If the two
strings are equal, then 0. Else, 1.
With most of these functions, they expect the strings to be readable and
not NULL but there is undefined behavior when you pass them NULL.
\subsection{\texorpdfstring{String Alteration: \keyword{strcpy}
\keyword{strcat}
\keyword{strdup}}{String Alteration: strcpy strcat strdup}}\label{string-alteration-strcpy-strcat-strdup}
\keyword{char\ *strcpy(char\ *dest,\ const\ char\ *src)} Copies the
string at \keyword{src} to \keyword{dest}. \textbf{assumes dest has enough
space for src}
\keyword{char\ *strcat(char\ *dest,\ const\ char\ *src)} Concatenates the
string at \keyword{src} to the end of destination. \textbf{This function
assumes that there is enough space for \keyword{src} at the end of
destination including the NULL byte}
\keyword{char\ *strdup(const\ char\ *dest)} Returns a \keyword{malloc}'ed
copy of the string.
\subsection{\texorpdfstring{String Search: \keyword{strchr}
\keyword{strstr}}{String Search: strchr strstr}}\label{string-search-strchr-strstr}
\keyword{char\ *strchr(const\ char\ *haystack,\ int\ needle)} Returns a
pointer to the first occurrence of \keyword{needle} in the
\keyword{haystack}. If none found, \keyword{NULL} is returned.
\keyword{char\ *strstr(const\ char\ *haystack,\ const\ char\ *needle)}
Same as above but this time a string!
\subsection{\texorpdfstring{String Tokenize:
\keyword{strtok}}{String Tokenize: strtok}}\label{string-tokenize-strtok}
A dangerous but useful function strtok takes a string and tokenizes it.
Meaning that it will transform the strings into separate strings. This
function has a lot of specs so please read the man pages a contrived
example is below.
\begin{code}[language=C]
#include <stdio.h>
#include <string.h>
int main(){
char* upped = strdup("strtok,is,tricky,!!");
char* start = strtok(upped, ",");
do{
printf("%s\n", start);
}while((start = strtok(NULL, ",")));
return 0;
}
\end{code}
\textbf{Output}
\begin{code}[language=C]
strtok
is
tricky
!!
\end{code}
What happens when I change \keyword{upped} like this?
\begin{code}[language=C]
char* upped = strdup("strtok,is,tricky,,,!!");
\end{code}
\section{\texorpdfstring{So what's a \keyword{struct}?}{So what's a struct?}}\label{so-whats-a-struct}
In low-level terms, a struct is just a piece of contiguous memory,
nothing more. Just like an array, a struct has enough space to keep all
of its members. But unlike an array, it can store different types.
Consider the contact struct declared above
\begin{code}[language=C]
struct contact {
char firstname[20];
char lastname[20];
unsigned int phone;
};
struct contact bhuvan;
\end{code}
\textbf{Brief aside}
\begin{code}[language=C]
/* a lot of times we will do the following typdef
so we can just write contact contact1 */
typedef struct contact contact;
contact bhuvan;
/* You can also declare the struct like this to get
it done in one statement */
typedef struct optional_name {
...
} contact;
\end{code}
If you compile the code without any optimizations and reordering, you
can expect the addresses of each of the variables to look like this.
\begin{code}[language=C]
&bhuvan // 0x100
&bhuvan.firstname // 0x100 = 0x100+0x00
&bhuvan.lastname // 0x114 = 0x100+0x14
&bhuvan.phone // 0x128 = 0x100+0x28
\end{code}
Because all your compiler does is say `hey reserve this much space, and
I will go and calculate the offsets of whatever variables you want to
write to'. The offsets are where the variable starts at. The phone variables starts
at the \keyword{0x128}th bytes and continues for sizeof(int) bytes, but
not always. \textbf{Offsets don't determine where the variable ends
though}. Consider the following hack that you see in a lot of kernel
code.
\begin{code}[language=C]
typedef struct {
int length;
char c_str[0];
} string;
const char* to_convert = "bhuvan";
int length = strlen(to_convert);
// Let's convert to a c string
string* bhuvan_name;
bhuvan_name = malloc(sizeof(string) + length+1);
/*
Currently, our memory looks like this with junk in those black spaces
___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___
bhuvan_name = | | | | | | | | | | | |
*/
bhuvan_name->length = length;
/*
This writes the following values to the first four bytes
The rest is still garbage
___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___
bhuvan_name = | 0 | 0 | 0 | 6 | | | | | | | |
*/
strcpy(bhuvan_name->c_str, to_convert);
/*
Now our string is filled in correctly at the end of the struct
___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ____
bhuvan_name = | 0 | 0 | 0 | 6 | b | h | u | v | a | n | \0 |
‾
*/
strcmp(bhuvan_name->c_str, "bhuvan") == 0 //The strings are equal!
\end{code}
\end{comment}
\section{Common Bugs}
ipc/images/440px-X86_Paging_4K.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

ipc/images/page_table1.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 20 KiB

ipc/ipc.bib
+10
View File
@@ -0,0 +1,10 @@
@manual{ibm709,
author = {International Business Machines Corporation (IBM)},
title = {IBM 709 Data Processing System Reference Manual},
date = {August 1958},
language = {English},
organization = {International Business Machines Corporation (IBM)},
pagetotal = {179},
pubstate = {August 1958},
url = {http://archive.computerhistory.org/resources/text/Fortran/102653991.05.01.acc.pdf}
}
ipc/ipc.md
+17 -17
View File
@@ -138,26 +138,26 @@ The numbers in the pipes are the file descriptors for each process and the arrow
## What is a pipe?
A POSIX pipe is almost like its real counterpart - you can stuff bytes down one end and they will appear at the other end in the same order. Unlike real pipes however, the flow is always in the same direction, one file descriptor is used for reading and the other for writing. The `pipe` system call is used to create a pipe.
```C
```
int filedes[2];
pipe (filedes);
printf("read from %d, write to %d\n", filedes[0], filedes[1]);
```
These file descriptors can be used with `read` -
```C
```
// To read...
char buffer[80];
int bytesread = read(filedes[0], buffer, sizeof(buffer));
```
And `write` -
```C
```
write(filedes[1], "Go!", 4);
```
## How can I use pipe to communicate with a child process?
A common method of using pipes is to create the pipe before forking.
```C
```
int filedes[2];
pipe (filedes);
pid_t child = fork();
@@ -169,7 +169,7 @@ if (child > 0) { /* I must be the parent */
```
The child can then send a message back to the parent:
```C
```
if (child == 0) {
write(filedes[1], "done", 4);
}
@@ -178,7 +178,7 @@ if (child == 0) {
Short answer: Yes, but I'm not sure why you would want to LOL!
Here's an example program that sends a message to itself:
```C
```
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
@@ -204,7 +204,7 @@ int main() {
The problem with using a pipe in this fashion is that writing to a pipe can block i.e. the pipe only has a limited buffering capacity. If the pipe is full the writing process will block! The maximum size of the buffer is system dependent; typical values from 4KB upto 128KB.
```C
```
int main() {
int fh[2];
pipe(fh);
@@ -222,7 +222,7 @@ int main() {
# Pipe Gotchas
Here's a complete example that doesn't work! The child reads one byte at a time from the pipe and prints it out - but we never see the message! Can you see why?
```C
```
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
@@ -257,7 +257,7 @@ The parent sends the bytes `H,i,(space),C...!` into the pipe (this may block if
The child starts reading the pipe one byte at a time. In the above case, the child process will read and print each character. However it never leaves the while loop! When there are no characters left to read it simply blocks and waits for more.
The call `putchar` writes the characters out but we never flush the `stdout` buffer. i.e. We have transferred the message from one process to another but it has not yet been printed. To see the message we could flush the buffer e.g. `fflush(stdout)` (or `printf("\n")` if the output is going to a terminal). A better solution would also exit the loop by checking for an end-of-message marker,
```C
```
while ((bytesread = read(fd[0], &buf, 1)) > 0) {
putchar(buf);
if (buf == '!') break; /* End of message */
@@ -273,7 +273,7 @@ At the C library level, C wraps these with a buffer and useful functions like pr
If you already have a file descriptor then you can 'wrap' it yourself into a FILE pointer using `fdopen` :
```C
```
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
@@ -292,7 +292,7 @@ However for pipes, we already have a file descriptor - so this is great time to
Here's a complete example using pipes that almost works! Can you spot the error? Hint: The parent never prints anything!
```C
```
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
@@ -315,7 +315,7 @@ int main() {
}
```
Note the (unnamed) pipe resource will disappear once both the child and parent have exited. In the above example the child will send the bytes and the parent will receive the bytes from the pipe. However, no end-of-line character is ever sent, so `fscanf` will continue to ask for bytes because it is waiting for the end of the line i.e. it will wait forever! The fix is to ensure we send a newline character, so that `fscanf` will return.
```C
```
change: fprintf(writer, "Score %d", 10 + 10);
to: fprintf(writer, "Score %d\n", 10 + 10);
```
@@ -341,7 +341,7 @@ Tip: Notice only the writer (not a reader) can use this signal.
To inform the reader that a writer is closing their end of the pipe, you could write your own special byte (e.g. 0xff) or a message ( `"Bye!"`)
Here's an example of catching this signal that does not work! Can you see why?
```C
```
#include <stdio.h>
#include <stdio.h>
#include <unistd.h>
@@ -415,7 +415,7 @@ Any `open` is called on a named pipe the kernel blocks until another process cal
## Race condition with named pipes.
What is wrong with the following program?
```C
```
//Program 1
int main(){
@@ -481,14 +481,14 @@ Another important aspect to note is the C files are **buffered** meaning that th
For files less than the size of a long, using fseek and ftell is a simple way to accomplish this:
Move to the end of the file and find out the current position.
```C
```
fseek(f, 0, SEEK_END);
long pos = ftell(f);
```
This tells us the current position in the file in bytes - i.e. the length of the file!
`fseek` can also be used to set the absolute position.
```C
```
fseek(f, 0, SEEK_SET); // Move to the start of the file
fseek(f, posn, SEEK_SET); // Move to 'posn' in the file.
```
@@ -499,7 +499,7 @@ See the man pages for fseek and ftell for more information.
## But try not to do this
**Note: This is not recommended in the usual case because of a quirk with the C language**. That quirk is that longs only need to be **4 Bytes big** meaning that the maximum size that ftell can return is a little under 2 Gigabytes (which we know nowadays our files could be hundreds of gigabytes or even terabytes on a distributed file system). What should we do instead? Use `stat`! We will cover stat in a later part but here is some code that will tell you the size of the file
```C
```
struct stat buf;
if(stat(filename, &buf) == -1){
return -1;
ipc/ipc.tex
+327 -706
View File
File diff suppressed because it is too large Load Diff
latexmkrc
+1
View File
@@ -0,0 +1 @@
add_cus_dep('glo', 'gls', 0, 'makeglo2gls'); sub makeglo2gls { system("makeindex -s '$_[0]'.ist -t '$_[0]'.glg -o '$_[0]'.gls '$_[0]'.glo"); }
main.pdf
BIN
View File
Binary file not shown.
main.tex
+8 -8
View File
@@ -23,7 +23,7 @@
\usepackage{chapterbib}
% Set up glossaries
\usepackage{glossaries}
\usepackage[toc,section=chapter]{glossaries}
\input{glossary.tex}
\makeglossaries
@@ -128,7 +128,7 @@ rulecolor=\color{shadecolor},}
% For pictures
\newenvironment{proof}{\begin{shaded*}\paragraph{Proof:}}{\hfill$\square$\end{shaded*}}
\newenvironment{aside}{\begin{shaded*}\textbf{Aside}\hfill}{\end{shaded*}}
\newenvironment{aside}{\begin{shaded*}\textbf{Aside}\\}{\end{shaded*}}
\newenvironment{tableau}{\begin{shaded*}\begin{longtable}}{\end{longtable}\end{shaded*}}
\usepackage{wrapfig}
@@ -151,18 +151,18 @@ rulecolor=\color{shadecolor},}
\end{multicols}
\include{introduction/introduction}
\include{introc/introc}
%\include{processes/processes}
%\include{malloc/malloc}
% \include{introc/introc}
% \include{processes/processes}
% \include{malloc/malloc}
%\include{threads/threads}
%\include{synchronization/synchronization}
\include{deadlock/deadlock}
%\include{scheduling/scheduling}
%\include{ipc/ipc}
% \include{ipc/ipc}
%\include{networking/networking}
%\include{filesystems/filesystems}
\include{signals/signals}
%\include{review/review}
%% \include{signals/signals}
% \include{review/review}
\include{appendix/appendix}
\printglossaries
malloc/malloc.bib
View File
malloc/malloc.md
+10 -10
View File
@@ -21,7 +21,7 @@ On typical architectures, the heap is part of the `Data segment` and starts just
## Do programs need to call brk or sbrk?
Not typically (though calling `sbrk(0)` can be interesting because it tells you where your heap currently ends). Instead programs use `malloc,calloc,realloc` and `free` which are part of the C library. The internal implementation of these functions will call `sbrk` when additional heap memory is required.
```C
```
void *top_of_heap = sbrk(0);
malloc(16384);
void *top_of_heap2 = sbrk(0);
@@ -32,7 +32,7 @@ Example output: `The top of heap went from 0x4000 to 0xa000`
## What is calloc?
Unlike `malloc`, `calloc` initializes memory contents to zero and also takes two arguments (the number of items and the size in bytes of each item). A naive but readable implementation of `calloc` looks like this:
```C
```
void *calloc(size_t n, size_t size)
{
size_t total = n * size; // Does not check for overflow!
@@ -52,7 +52,7 @@ An advanced discussion of these limitations is [here](http://locklessinc.com/art
Programmers often use `calloc` rather than explicitly calling `memset` after `malloc`, to set the memory contents to zero. Note `calloc(x,y)` is identical to `calloc(y,x)`, but you should follow the conventions of the manual.
```C
```
// Ensure our memory is initialized to zero
link_t *link = malloc(256);
memset(link, 0, 256); // Assumes malloc returned a valid address!
@@ -64,7 +64,7 @@ If the operating system did not zero out contents of physical RAM it might be po
Unfortunately this means that for `malloc` requests before any memory has been freed and simple programs (which end up using newly reserved memory from the system) the memory is _often_ zero. Then programmers mistaken write C programs that assume malloc'd memory will _always_ be zero.
```C
```
char* ptr = malloc(300);
// contents is probably zero because we get brand new memory
// so beginner programs appear to work!
@@ -79,7 +79,7 @@ Performance! We want malloc to be as fast as possible. Zeroing out memory may be
## What is realloc and when would you use it?
`realloc` allows you to resize an existing memory allocation that was previously allocated on the heap (via malloc,calloc or realloc). The most common use of realloc is to resize memory used to hold an array of values. A naive but readable version of realloc is suggested below
```C
```
void * realloc(void * ptr, size_t newsize) {
// Simple implementation always reserves more memory
// and has no error checking
@@ -91,7 +91,7 @@ void * realloc(void * ptr, size_t newsize) {
}
```
An INCORRECT use of realloc is shown below:
```C
```
int *array = malloc(sizeof(int) * 2);
array[0] = 10; array[1] = 20;
// Ooops need a bigger array - so use realloc..
@@ -101,7 +101,7 @@ array[2] = 30;
The above code contains two mistakes. Firstly we needed 3*sizeof(int) bytes not 3 bytes.
Secondly realloc may need to move the existing contents of the memory to a new location. For example, there may not be sufficient space because the neighboring bytes are already allocated. A correct use of realloc is shown below.
```C
```
array = realloc(array, 3 * sizeof(int));
// If array is copied to a new location then old allocation will be freed.
```
@@ -117,7 +117,7 @@ Very! Allocating and de-allocating heap memory is a common operation in most app
## What is the silliest malloc and free implementation and what is wrong with it?
```C
```
void* malloc(size_t size)
// Ask the system for more bytes by extending the heap space.
// sbrk Returns -1 on failure
@@ -248,7 +248,7 @@ This gets very sinister when you implementing coalescing and splitting (next sec
When `free` is called we need to re-apply the offset to get back to the 'real' start of the block (remember we didn't give the user a pointer to the actual start of the block?), i.e. to where we stored the size information.
A naive implementation would simply mark the block as unused. If we are storing the block allocation status in the lowest size bit, then we just need to clear the bit:
```C
```
*p = (*p) & ~1; // Clear lowest bit
```
However, we have a bit more work to do: If the current block and the next block (if it exists) are both free we need to coalesce these blocks into a single block.
@@ -309,7 +309,7 @@ The precise layout of the stack's contents and order of the automatic variables
The example below demonstrates how the return address is stored on the stack. For a particular 32 bit architecture [Live Linux Machine](http://cs-education.github.io/sys/), we determine that the return address is stored at an address two pointers (8 bytes) above the address of the automatic variable. The code deliberately changes the stack value so that when the input function returns, rather than continuing on inside the main method, it jumps to the exploit function instead.
````C
````
// Overwrites the return address on the following machine:
// http://cs-education.github.io/sys/
#include <stdio.h>
malloc/malloc.tex
+214 -625
View File
File diff suppressed because it is too large Load Diff