From 9db30a7c1dae04e05782f6778712c03cef1afbe7 Mon Sep 17 00:00:00 2001 From: Geoffrey Challen Date: Fri, 26 Dec 2014 16:37:52 -0500 Subject: [PATCH] New. --- certainty.tex | 87 +++++++++++++++++++++++++++++++++++++++++++-------------------------------------------- include/start.tex | 4 ++-- 2 files changed, 45 insertions(+), 46 deletions(-) diff --git a/certainty.tex b/certainty.tex index 25f59d2..9f3bf9f 100644 --- a/certainty.tex +++ b/certainty.tex @@ -2,10 +2,10 @@ \label{sec-certainty} While \texttt{maybe} allows programmers to specify multiple alternatives, -ultimately at runtime only one alternative can be executed. Either a single, -globally-optimal alternative must be determined, or a deterministic decision -procedure must be developed. Before discussing options for \textit{adapting} -an app to its runtime environment, we first explain our runtime's support for +ultimately only one alternative can be executed at runtime. Either a single, +globally-optimal alternative must be identified, or a deterministic decision +procedure must be developed. Before discussing options for adapting an app to +its runtime environment, we first explain our runtime's support for \texttt{maybe} alternatives, including \textit{a posteriori} evaluation and data collection. Then, we discuss how \texttt{maybe} testing enables a variety of different adaptation patterns. @@ -39,11 +39,11 @@ automatically-generated labels to aid this process. If a \texttt{maybe} alternative throws an error, the system will bypass the \texttt{evaluate} block and give it the worst possible score. By integrating a form of record-and-replay~\cite{gomez2013reran}, it may be possible to roll -back the failed alternative and retry another. \texttt{maybe} is intended to -enable adaptation, not avoid errors, but the existence of other alternatives -provides a way to work around failures caused by uncertainty. Fault tolerance -may also encourage developers to use \texttt{maybe} statements to prototype -alternatives to existing well-tested code. +back the failed alternative and retry another. While \texttt{maybe} is +intended to enable adaptation, not avoid errors, the existence of other +alternatives provides a way to work around failures caused by uncertainty. +Fault tolerance may also encourage developers to use \texttt{maybe} +statements to prototype alternatives to existing well-tested code. A final question concerns when a \texttt{maybe} alternative should be evaluated. Some alternatives may require evaluation immediately after @@ -69,8 +69,8 @@ ways for the developer to guide and control any step in the process. \subsubsection{Runtime control} To begin, we briefly outline how our Android prototype implements the -\texttt{maybe} statement. We (1) rewrite each \texttt{maybe} statement to an -\texttt{if-else} statement controlled by a call into the \texttt{maybe} +\texttt{maybe} statement. We (1) rewrite each \texttt{maybe} conditional to +an \texttt{if-else} statement controlled by a call into the \texttt{maybe} system and (2) generate a similar setter for each \texttt{maybe} variable. Variable values and code branches are now all under the control of a separate \texttt{maybe} service which can be deployed as a separate app or @@ -100,8 +100,8 @@ simulation environments that accurately reflect all of the uncertainties inherent to mobile systems programming, however, is difficult. To complicate matters, \texttt{maybe} alternatives may depend on details of user interaction that are difficult to know \textit{a priori}, particularly when -new apps or functionalities are being investigated. So in most cases we -believe post-deployment testing will be required. +new apps or features are being developed. So in most cases we believe +post-deployment testing will be required. However, pre-deployment testing may still be a valuable approach, particularly when a large number of \texttt{maybe} statements are being used. @@ -114,16 +114,15 @@ significant impact on performance and should be evaluated first. Other Eventually code containing a number of \texttt{maybe} statements will be deployed on thousands or millions of devices. At this point, large-scale -split testing and learning can begin. If the user community is large enough, -then it may be possible to collect statistically-significant results even for -all possible permutations of \texttt{maybe} alternatives. For apps with a -small number of users, or comparatively large number of \texttt{maybe} -statements, we can collect data for variations of one \texttt{maybe} -statement while holding the others constant. As an adaptation policy is +split testing and data-driven learning can begin. If the user community is +large enough, it may be possible to collect statistically-significant results +even for all possible permutations of \texttt{maybe} alternatives. For apps +with a small number of users, or a large number of \texttt{maybe} statements, +we can collect data for variations of one or several \texttt{maybe} +statements while holding the rest constant. As an adaptation policy is designed and deployed for the statement being tested, we begin to vary and -measure the next \texttt{maybe} statement. We will provide a web interface -allowing the developer to determine which \texttt{maybe} statements should be -tested at any given time. +measure the next group of \texttt{maybe} statements. Developers can observe +and control the testing process through a web interface. Each time a \texttt{maybe} statement is reached or \texttt{maybe} variable is set, the \texttt{maybe} system records: @@ -145,7 +144,7 @@ and platform information; networking provider and conditions; location; battery level; and so on. \end{itemize} -% + This dataset is periodically uploaded to the \texttt{maybe} server and used to drive the adaptation approaches discussed next. @@ -154,17 +153,17 @@ to drive the adaptation approaches discussed next. While large-scale split testing is intended to provide good coverage over all possible sources of uncertainty we have discussed, it still normally requires that only one choice be made at any given time---implying that two -alternatives may \textit{never} be evaluated under identical conditions. For +alternatives may never be evaluated under identical conditions. For \texttt{maybe} statements, however, we are exploring the idea of performing \textit{simultaneous} split testing. In this model the app forks at the top -of the \texttt{maybe} statement, executes and scores all alternatives, and then -continues with the outputs from the best alternative at the bottom of the -\texttt{maybe} statement. On single-core devices this can be done in serial, -while the growing number of multi-core smartphones provides the option of -doing this in parallel. The benefit of this approach is that each alternative -is executed under near-identical conditions. The drawbacks include the -overhead of the redundant executions and the possibility for interference -between alternatives executing in parallel. +of the \texttt{maybe} statement, executes and scores all alternatives, and +then continues with the outputs from the best alternative at the bottom of +the \texttt{maybe} statement. On single-core devices this can be done in +serial, while the growing number of multi-core smartphones provides the +option of doing this in parallel. The benefit of this approach is that each +alternative is executed under near-identical conditions. The drawbacks +include the overhead of the redundant executions and the possibility for +interference between alternatives executing in parallel. \subsection{\texttt{\large maybe} Endgames} @@ -206,11 +205,10 @@ second, somewhat easier case. If the alternative is determined through static adaptation then the correct choice is a function of some unchanging (or very-slowly changing) aspect of the deployed environment. Examples include the device model, average network -conditions, the other apps installed on the device, or some slowly-changing -user characteristic such as gender, age, or charging habits. In this case it -is possible that the correct alternative can be determined through some -clustering based on these features, and once determined will remain the best -choice over long time intervals. +conditions, the other apps installed on the device, or user characteristics +such as gender, age, or charging habits. In this case it is possible that the +correct alternative can be determined through clustering based on these +features, and once determined will remain the best choice for a long time. \subsubsection{Dynamic adaptation} @@ -225,13 +223,14 @@ Note that \texttt{evaluate} blocks are \textit{not} intended to accomplish this kind of adaptation. First, they run after the \texttt{maybe} statement has been executed, not before. Second, per-\texttt{maybe} strategy defeats the flexibility inherent to the \texttt{maybe} approach and would devolve -into the fragile decision-making we are trying to avoid. Instead, the -\texttt{maybe} system allows developers to experiment with and evaluate a -variety of different dynamic adaptation strategies deployed in a companion -library, with the choice guided by post-deployment testing. For example, if -the performance of an alternative is discovered to be correlated with a link -providing a certain amount of bandwidth, then that adaptation strategy can be -connected with that particular \texttt{maybe} statement. +into the fragile decision-making we are trying to avoid. + +Instead, the \texttt{maybe} system allows developers to experiment with and +evaluate a variety of different dynamic adaptation strategies deployed in a +companion library, with the choice guided by post-deployment testing. For +example, if the performance of an alternative is discovered to be correlated +with a link providing a certain amount of bandwidth, then that adaptation +strategy can be connected with that particular \texttt{maybe} statement. Observe that in some cases of dynamic adaptation, what begins as a \texttt{maybe} statement may end as effectively \texttt{if-else} statement diff --git a/include/start.tex b/include/start.tex index 835059a..fd807b8 100644 --- a/include/start.tex +++ b/include/start.tex @@ -27,8 +27,8 @@ \usepackage[all]{hypcap} -\setlist[itemize]{leftmargin=*,topsep=2pt,itemsep=-0.5ex,partopsep=1ex,parsep=1ex} -\setlist[enumerate]{leftmargin=*,topsep=2pt,itemsep=-0.5ex,partopsep=1ex,parsep=1ex} +\setlist[itemize]{leftmargin=*,itemsep=-0.1ex} +%\setlist[enumerate]{leftmargin=*,topsep=0.5ex,itemsep=0.5ex,partopsep=0.5ex,parsep=0.5ex} \paperheight 11in \paperwidth 8.5in -- libgit2 0.22.2