From e9c9a29bbb2dfb4bd84ad35d582e044a8feec7bb Mon Sep 17 00:00:00 2001 From: Geoffrey Challen Date: Tue, 23 Dec 2014 11:01:48 -0500 Subject: [PATCH] Certainty section. --- certainty.tex | 88 ++++++++++++++++++++++++++++++++++++++++------------------------------------------------ 1 file changed, 40 insertions(+), 48 deletions(-) diff --git a/certainty.tex b/certainty.tex index 0db24a5..f9d8bfb 100644 --- a/certainty.tex +++ b/certainty.tex @@ -86,7 +86,7 @@ ways for the developer to guide and control any step in the process. \subsubsection{Runtime control} To begin, we briefly outline how our Android prototype implements the -\texttt{maybe} statement. We (1) rewrite each \texttt{maybe} code block to an +\texttt{maybe} statement. We (1) rewrite each \texttt{maybe} statement to an \texttt{if-else} statement controlled by a call into the \texttt{maybe} system and (2) generate a similar setter for each \texttt{maybe} variable. Variable values and code branches are now all under the control of a separate @@ -120,13 +120,12 @@ are difficult to know \textit{a priori}, particularly when new apps or functionalities are being investigated. So in most cases we believe post-deployment testing will be required. -However, pre-deployment testing may still be a valuable approach, particularly -when a large amount of uncertainties are present and a correspondingly-large -number of \texttt{maybe} statements are being used. Since this can explode -the adaptation space, simulation may be able to help guide the developer's -choices of which \texttt{maybe} statements may have a significant impact on -performance and should be assessed first. Other \texttt{maybe} statements -can be assessed later or eliminated altogether. +However, pre-deployment testing may still be a valuable approach, +particularly when a large number of \texttt{maybe} statements are being used. +Since this can explode the adaptation space, simulations may be able to help +guide the developer's choices of which \texttt{maybe} statements may have a +significant impact on performance and should be assessed first. Other +\texttt{maybe} statements can be assessed later or eliminated. \subsubsection{Split testing} @@ -164,9 +163,8 @@ battery level; and so on. \end{itemize} % -This dataset is periodically uploaded to the \texttt{maybe} server by the -\texttt{maybe} service and used to drive the adaptation approaches discussed -next. +This dataset is periodically uploaded to the \texttt{maybe} server and used +to drive the adaptation approaches discussed next. \subsubsection{Simultaneous split testing} @@ -174,24 +172,23 @@ While large-scale split testing is intended to provide good coverage over all possible sources of uncertainty we have discussed, it still normally requires that only one choice be made at any given time---implying that two alternatives may \textit{never} be evaluated under identical conditions. For -\texttt{maybe} code blocks, however, we are exploring the idea of performing +\texttt{maybe} statements, however, we are exploring the idea of performing \textit{simultaneous} split testing. In this model the app forks at the top -of the \texttt{maybe} block, executes and scores all alternatives, and then +of the \texttt{maybe} statement, executes and scores all alternatives, and then continues with the outputs from the best alternative at the bottom of the \texttt{maybe} statement. On single-core devices this can be done in serial, while the growing number of multi-core smartphones provides the option of doing this in parallel. The benefit of this approach is that each alternative is executed under near-identical conditions. The drawbacks include the overhead of the redundant executions and the possibility for interference -between alternatives executing in parallel. However, we are excited to -explore this option in our prototype. +between alternatives executing in parallel. \subsection{\texttt{\large maybe} Endgames} The entire \texttt{maybe} approach is predicated on the fact that there does exist, among the alternatives, a right choice, even if it depends on many factors and uncertainties. We continue by discussing how the dataset -generated by post-deployment testing can be consumed to determine how to +generated by post-deployment testing can be used to determine how to correctly choose \texttt{maybe} alternatives at runtime. \subsubsection{Simple cases} @@ -203,11 +200,10 @@ testing of that alternative and even automatically rewrite that portion of code to remove the \texttt{maybe} statement. However, it is also possible that the situation may change in the future when a new device, or Android version, or battery technology is introduced, and so the programmer may also -choose to preserve the alternatives to enable continuous adaptation as -described in Section~\ref{subsec-continuous}. +choose to preserve the flexibility in case it is useful in the future. The slightly more complicated case is when testing reveals that alternatives -provide stable tradeoffs between energy and performance---one block always +provide stable tradeoffs between energy and performance---one alternative always saves energy at the cost of performance. In this case the system only has to determine whether to prioritize energy or performance. While this decision seems simple, it is itself complicated by differences in battery capacity, @@ -226,11 +222,10 @@ second, somewhat easier case. If the alternative is determined through static adaptation then the correct choice is a function of some unchanging (or very-slowly changing) aspect of -the deployed environment. Examples include the model of the device, overall -network conditions in the country in which the device is being used, the -other apps installed on the device, or some slowly-changing user -characteristic such as gender, age, or charging habits. In this case it is -possible that the correct alternative can be determined through some +the deployed environment. Examples include the device model, average network +conditions, the other apps installed on the device, or some slowly-changing +user characteristic such as gender, age, or charging habits. In this case it +is possible that the correct alternative can be determined through some clustering based on these features, and once determined will remain the best choice over long time intervals. @@ -243,37 +238,34 @@ single alternative can be chosen even for a single user. Instead, the \texttt{maybe} system allows developers to evaluate one or more strategies to drive the runtime alternative selection process. -Note that \texttt{evaluate} blocks are \textit{not} intended to accomplish this -kind of adaptation. First, they run after the \texttt{maybe} block has been -executed, not before. Second, a single strategy defeats the flexibility -inherent to the \texttt{maybe} approach and would devolve into the same -fragile decision-making logic we are trying to avoid. - -Instead, the \texttt{maybe} system allows developers to experiment with and -evaluate a variety of different dynamic adaptation strategies deployed in a -companion library, with the choice guided by post-deployment testing. For -example, if the performance of an alternative is discovered to be correlated -with the presence of a network link with over a certain bandwidth, then that -adaptation strategy can be connected with that particular \texttt{maybe} -block. +Note that \texttt{evaluate} blocks are \textit{not} intended to accomplish +this kind of adaptation. First, they run after the \texttt{maybe} statement +has been executed, not before. Second, per-\texttt{maybe} strategy defeats +the flexibility inherent to the \texttt{maybe} approach and would devolve +into the fragile decision-making we are trying to avoid. Instead, the +\texttt{maybe} system allows developers to experiment with and evaluate a +variety of different dynamic adaptation strategies deployed in a companion +library, with the choice guided by post-deployment testing. For example, if +the performance of an alternative is discovered to be correlated with a link +providing a certain amount of bandwidth, then that adaptation strategy can be +connected with that particular \texttt{maybe} statement. Observe that in some cases of dynamic adaptation, what begins as a -\texttt{maybe} statement may end as effectively \texttt{if-else} block -switching on some of the same static thresholds that we attacked to motivate +\texttt{maybe} statement may end as effectively \texttt{if-else} statement +switching on a static threshold---the same approach we attacked to motivate our system. However, through the process of arriving at this point we have -determined several things that were initially unknown: What the alternatives +determined several things that were initially unknown: what the alternatives accomplish, that a single threshold works for all users, and what that -threshold is. And if the developer chooses to maintain that statement as a -\texttt{maybe} block, they can continue to perform testing and modify their -adaptation strategy as devices and users change. +threshold is. And by maintaining the choice as a \texttt{maybe} statement, +they can continue the adaptation process as devices, users, and networks +change. Another benefit of this approach is that time-varying decisions can be outsourced to developers with expertise in the particular area driving -adaptation policy decisions. For example, if the developer chooses to make a -energy-performance tradeoff dynamically based on the energy conditions at -that time, they can connect that \texttt{maybe} statement to a sophisticated -machine learning algorithm written by an expert in energy adaptation, instead -of being forced to implement their own simplistic approach. +adaptation policy decisions. For example, by exposing an energy-performance +tradeoff through a \texttt{maybe} statement, a developer allows it to be +driven by a sophisticated machine learning algorithm written by an expert in +energy adaptation, instead of their own ad-hoc approach. \subsubsection{Manual adaptation} -- libgit2 0.22.2