From bbab019478f7a71aceceebf76e9dca80e1fad47e Mon Sep 17 00:00:00 2001 From: Geoffrey Challen Date: Mon, 22 Dec 2014 17:47:51 -0500 Subject: [PATCH] Working. --- .ispell_english | 7 +++++++ certainty.tex | 179 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------------------------------------------------------------------------------- discussion.tex | 41 +++++++++++++++++++++++++++++++++++++++++ introduction.tex | 4 +++- maybe.tex | 87 +++++++++++++++++++++++---------------------------------------------------------------- paper.tex | 1 + 6 files changed, 173 insertions(+), 146 deletions(-) create mode 100644 discussion.tex diff --git a/.ispell_english b/.ispell_english index 3e9971c..09f14aa 100644 --- a/.ispell_english +++ b/.ispell_english @@ -2,4 +2,11 @@ Android APIs app apps +Endgames +JSON +outsourced +pre +runtime's +smartphone +smartphones Wifi diff --git a/certainty.tex b/certainty.tex index e9ab22f..5854930 100644 --- a/certainty.tex +++ b/certainty.tex @@ -2,85 +2,88 @@ \label{sec-certainty} While \texttt{maybe} allows programmers to specify multiple alternatives, -ultimately only one alternative may be used. Either a single, globally best -alternative must be chosen, or a deterministic decision procedure must be -selected. Before discussing options for \textit{adapting} an app to its -runtime environment, we first explain our runtime's support for +ultimately at runtime only one alternative can be executed. Either a single, +globally-optimal alternative must be determined, or a deterministic decision +procedure must be developed. Before discussing options for \textit{adapting} +an app to its runtime environment, we first explain our runtime's support for \texttt{maybe} alternatives, including \textit{a posteriori} evaluation and -data-collection. Then, we discuss how \texttt{maybe} testing enables a +data collection. Then, we discuss how \texttt{maybe} testing enables a variety of different adaptation patterns. -\subsection{Evaluating Alternatives} The optional \texttt{evaluate} block of -a \texttt{maybe} statement allows programmers to provide app-specific -\textit{a posteriori} evaluation logic. However, in many cases, we expect -that \texttt{maybe} statements will be used to achieve common objectives such -as improving performance or saving energy. To steamline application -development, our current system assesses \texttt{maybe} statements without a -\texttt{better} block in terms of both energy and performance. In cases where -one alternative is a clear winner in terms of both energy and performance, -that alternative will be used---although note that this selection may still -need to be time-varying since it may depend on exogenous factors. - -Cases where \texttt{maybe} blocks provide an energy-performance tradeoff -require more investigation to handle. We are exploring several options, -including collapsing both metrics into a single score by computing the -energy-delay product of each alternative, or allowing users to set a single -per-app preference indicating whether energy or performance should determine -\texttt{maybe} alternative selection. - -When \texttt{maybe} statements include a \texttt{better} block, their custom -evaluation logic is expected to merge any relevant factors into a single -score, although \texttt{better} blocks can still record other relevant -information to aid in understanding app behavior. Specifically, -\texttt{better} blocks return a JSON object where the ``score'' key is used -as the score but all other keys are recorded for later analysis. Our standard -energy and performance \texttt{better} block records energy and performance +\subsection{Evaluating Alternatives} + +The optional \texttt{evaluate} block of a \texttt{maybe} statement allows +programmers to provide app-specific \textit{a posteriori} evaluation logic. +However, in many cases, we expect that \texttt{maybe} statements will be used +to achieve common objectives such as improving performance or saving energy. +To streamline application development, our current system evaluates +\texttt{maybe} statements without a \texttt{evaluate} block by measuring both +energy and performance. In cases where one alternative optimizes both, that +alternative will be used---although the decision may still be time-varying +due to dependence on time-varying factors such as network availability. + +\sloppypar{Cases where \texttt{maybe} statements provide an +energy-performance tradeoff require more investigation. We are exploring +several options, including collapsing both metrics into a single score by +computing the energy-delay product (EDP) of each alternative, or allowing +users to set a per-app preference indicating whether energy or performance +should determine \texttt{maybe} alternative selection.} + +When \texttt{maybe} statements include a \texttt{evaluate} block, their +custom evaluation logic must merge any relevant factors into a single score, +although \texttt{evaluate} blocks can still record other relevant information +to aid in understanding app behavior. Specifically, while the \texttt{score} +value is used to evaluate the alternative, the entire JSON object returned by +the \texttt{evaluate} block is recorded for later analysis. Our standard +energy and performance \texttt{evaluate} block records energy and performance separately in the JSON object but combines them as required into a single -score. Note that because the \texttt{better} block delivers information to +score. Note that because the \texttt{evaluate} block delivers information to the developer through the \texttt{maybe} service, as described below, \texttt{maybe} statements can be connected with end-to-end app performance -metrics that would normally not be visible on the device. \texttt{better} -blocks may also want to query the user directly, and we are considering ways -to adjust their semantics to make this possible. - -In some cases we expect that \texttt{better} blocks may need to know which -alternative was executed in order to determine the score---for example, if -the two alternatives produce different quality output while trading off -performance or energy consumption. The simplest solution is to have each -alternative set a variable indicating that it was used, but we believe that -eventually including some form of labeling syntax for \texttt{maybe} -alternatives may be beneficial. - -As a side note, if a \texttt{maybe} alternative encounters an error, the -system will automatically both retry another alternative and give the -error-generating alternative the worst possible score. In this case any -custom evaluation logic will not be executed for the failing alternative. -While \texttt{maybe} is not designed as a way to avoid errors, the existence -of other alternatives provides our system with a way to work around -failures sometimes caused by environmental factors. - -A final question concerns when a \texttt{maybe} alternative should be assessed. -In some -cases it may be appropriate for post-mortem evaluation to happen immediately after execution. -In others, it may be necessary to execute the same alternative over a period -of time to perform a fair comparison. As described previously, -\texttt{better} blocks can indicate explicitly whether or not to continue -evaluating the alternative, and we are determining how to make a similar -choice available to \texttt{maybe} statements that do not use customized -evaluation logic. In both cases, however, the \texttt{maybe} system allows -developers continuous per-statement control over alternative evaluation and -selection as described in more detail later in this section. +metrics that would normally not be visible on the device. \texttt{evaluate} +blocks may also want to query the user directly, and we are exploring ways to +make this possible. + +In some cases we expect that \texttt{evaluate} blocks may need to know which +alternative was executed to determine the score---for example, if the two +alternatives produce different quality output. The simplest solution is to +have each alternative set a variable indicating that it was used, but we +believe that eventually including some form of labeling syntax for +\texttt{maybe} alternatives may be beneficial. + +If a \texttt{maybe} alternative throws an error, the system will bypass the +\texttt{evaluate} block and give it the worst possible score. By integrating +a form of record-and-replay~\cite{FIXME}, it may also be possible to roll +back the failed alternatives changes and retry another alternative. +\texttt{maybe} is intended to enable adaptation, not avoid errors, but the +existence of other alternatives provides our system with a way to work around +failures caused by uncertainty. Resistance to errors may also encourage +developers to use \texttt{maybe} statements to prototype alternatives to +existing well-tested code. + +A final question concerns when a \texttt{maybe} alternative should be +assessed. In some cases it may be appropriate for \textit{a posteriori} +evaluation to happen immediately after execution. In others, it may be +necessary to execute the same alternative over a period of time to perform a +fair comparison. As described previously, \texttt{evaluate} blocks can +indicate explicitly whether or not to continue evaluating the alternative, +and we are determining how to make a similar choice available to +\texttt{maybe} statements that do not use customized evaluation logic. In +addition, \texttt{evaluate} blocks can store state across multiple +alternative executions allowing them to evaluate not only micro- but also +macro-level decisions. In both cases, however, the \texttt{maybe} system +allows developers continuous per-statement control over alternative +evaluation and selection as described in more detail later in this section. \subsection{\texttt{\large maybe} Alternative Testing} -We next describe the pre- and post-deployment testing that helps developers to -design an \textit{adaptation} policy, a strategy for ultimately selecting -between alternatives. While the \texttt{maybe} system -automates many of the tedious tasks normally associated with large-scale -testing, we still provide ways for the developer to guide and even override -any step in the process. +We next describe the pre- and post-deployment testing that helps developers +to design an \textit{adaptation} policy, a strategy for ultimately selecting +between alternatives. While the \texttt{maybe} system automates many of the +tedious tasks normally associated with large-scale testing, we still provide +ways for the developer to guide and control any step in the process. -\subsubsection{Runtime Control} +\subsubsection{Runtime control} To begin, we briefly outline how our Android prototype system implements the \texttt{maybe} construct. We (1) rewrite each \texttt{maybe} code block to an @@ -108,7 +111,7 @@ so by periodically resetting the value using another \texttt{maybe} statement. This ensures that \texttt{maybe} variables only change when the developer expects them to. -\subsubsection{Simulation or Emulation} +\subsubsection{Simulation or emulation} Pre-deployment simulation or emulation may provide a way to efficiently assess \texttt{maybe} blocks without involving users. Building simulation @@ -127,7 +130,7 @@ choices of which \texttt{maybe} statements may have a significant impact on performance and should be assessed first. Other \texttt{maybe} statements can be assessed later or eliminated altogether. -\subsubsection{Split Testing} +\subsubsection{Split testing} Eventually code containing a number of \texttt{maybe} statements will be deployed on thousands or millions of devices. At this point, large-scale @@ -150,10 +153,10 @@ set, the \texttt{maybe} system records: \item what \texttt{maybe} was reached; \item what alternative was used and why. This includes all environmental -features used to make the selection, as well as any other available +features used to make the decision, as well as any other available provenance information; -\item what \texttt{better} block evaluated the alternative, and the entire +\item what \texttt{evaluate} block evaluated the alternative, and the entire JSON object it returned, including the score; \item and a variety of other environmental and configuration parameters @@ -167,7 +170,7 @@ This dataset is periodically uploaded to the \texttt{maybe} server by the \texttt{maybe} service and used to drive the adaptation approaches discussed next. -\subsubsection{Simultaneous Split Testing} +\subsubsection{Simultaneous split testing} While large-scale split testing is intended to provide good coverage over all possible sources of uncertainty we have discussed, it still normally requires @@ -193,7 +196,7 @@ factors and uncertainties. We continue by discussing how the dataset generated by post-deployment testing can be consumed to determine how to correctly choose \texttt{maybe} alternatives at runtime. -\subsubsection{Simple Cases} +\subsubsection{Simple cases} In the simplest case, testing may reveal that a single alternative performs the best on all devices, for all users, at all times. In this situation, the @@ -202,8 +205,8 @@ testing of that alternative and even automatically rewrite that portion of code to remove the \texttt{maybe} statement. However, it is also possible that the situation may change in the future when a new device, or Android version, or battery technology is introduced, and so the programmer may also -choose to preserve the alternatives to enable later retesting and possible -adaptation. +choose to preserve the alternatives to enable continuous adaptation as +described in Section~\ref{subsec-continuous}. The slightly more complicated case is when testing reveals that alternatives provide stable tradeoffs between energy and performance---one block always @@ -215,7 +218,7 @@ each user. However, the stability of the alternatives' outcomes means that once an energy or performance policy decision has been made, the choice of alternative has also been made. -\subsubsection{Static Adaptation} +\subsubsection{Static adaptation} In the more complicated cases, testing reveals that the choice of alternative depends on some subset of the factors driving uncertainty in mobile systems @@ -233,7 +236,7 @@ possible that the correct alternative can be determined through some clustering based on these features, and once determined will remain the best choice over long time intervals. -\subsubsection{Dynamic Adaptation} +\subsubsection{Dynamic adaptation} If the choice of alternative depends on dynamic factors such as the accuracy of location services, the amount of energy left in the battery, or the type @@ -242,7 +245,7 @@ single alternative can be chosen even for a single user. Instead, the \texttt{maybe} system allows developers to evaluate one or more strategies to drive the runtime alternative selection process. -Note that \texttt{better} blocks are \textit{not} intended to accomplish this +Note that \texttt{evaluate} blocks are \textit{not} intended to accomplish this kind of adaptation. First, they run after the \texttt{maybe} block has been executed, not before. Second, a single strategy defeats the flexibility inherent to the \texttt{maybe} approach and would devolve into the same @@ -274,7 +277,7 @@ that time, they can connect that \texttt{maybe} statement to a sophisticated machine learning algorithm written by an expert in energy adaptation, instead of being forced to implement their own simplistic approach. -\subsubsection{Manual Adaptation} +\subsubsection{Manual adaptation} In some cases even our best efforts to automatically adapt may fail, and it may be impossible to predict which alternative is best for a particular user @@ -285,3 +288,17 @@ are significant then the \texttt{maybe} alternatives may need to be exposed to the user through a settings menu. Fortunately, information obtained through testing can still be presented to the user to guide their decision. Note that this requires labeling alternatives in a human-readable way. + +\subsection{Continuous Adaptation} +\label{subsec-continuous} + +Finally, even once a decision process for a particular \texttt{maybe} +alternative has been developed, it should be periodically revisited as users, +devices, networks, batteries, and other factors affecting mobile apps +continue to change. To enable continuous adaptation, developers can configure +\texttt{maybe} statements to continue to periodically experiment with +alternatives other than the one selected by the alternative testing process. +Changes in alternative performance relative to the expectations established +during the last round of alternative testing may trigger a large-scale +reexamination of that \texttt{maybe} statement using the same process +described above. diff --git a/discussion.tex b/discussion.tex new file mode 100644 index 0000000..c4ed7ae --- /dev/null +++ b/discussion.tex @@ -0,0 +1,41 @@ +\section{Discussion} +\label{sec-discussion} + +We have yet to determine how well programmers will be able to use the +\texttt{maybe} statement. Structurally \texttt{maybe} statements are similar +to the ubiquitous \texttt{if-else} construct in that at the bottom of the +statement the programmer may not be sure which block was executed. However, +while with \texttt{if-else} blocks it can be determined which block was +executed by examining the branch conditions, \texttt{maybe} statements +require developers to record which alternative was executed if downstream +code depends on the decision. To coordinate the adaptation of multiple code +paths a single \texttt{maybe} variable---such as the policy string in +Figure~\ref{fig-maybeexamples}---can be used to control multiple +\texttt{if-else} statements. + +While \texttt{maybe} is a powerful programming tool, it is important that it +be used sparingly. If objective dependencies exist between \texttt{maybe} +statements, the overall configuration space may expand exponentially, +complicating the process of determining the best alternatives described in +Section~\ref{sec-certainty}. Compile-time analysis may be required to detect +dependencies between \texttt{maybe} statements and ensure that downstream +optimization remains feasible. + +One important place where \texttt{maybe} should not be used is when +adaptation is not app-specific and can be refactored into a library serving +multiple apps. As an example, an app should not use \texttt{maybe} to decide +which network interface to use when attempting to achieve a common objective, +such as maximizing throughput. This choice would be better refactored into a +dedicated library, which might use its own internal \texttt{maybe} +statements. Not only is the resulting codebase smaller, but the total number +of \texttt{maybe} statements that the system has to determine how to handle +is reduced. + +However, the \texttt{maybe} statement represents a fundamentally different +approach from previously attempts to enable runtime adaptation that relied on +libraries. First, libraries + +Finally, note that structured uncertainty is not randomness. The +\texttt{maybe} statement indicates that during any given execution one +alternative may be better than the others---even if the developer or system +are not sure which alternative to use. diff --git a/introduction.tex b/introduction.tex index 67cf855..3ee0f1c 100644 --- a/introduction.tex +++ b/introduction.tex @@ -88,7 +88,9 @@ maybe { \vspace*{-0.1in} \caption{\textbf{Example \texttt{maybe} Statement.} The programmer provides -multiple equivalent alternatives.} +multiple equivalent alternatives. The system determines how to choose between +them at runtime.} + \label{fig-example-maybe} \vspace*{-0.1in} diff --git a/maybe.tex b/maybe.tex index 047f686..fb2d66f 100644 --- a/maybe.tex +++ b/maybe.tex @@ -1,10 +1,10 @@ \section{\texttt{\large maybe} Statement Semantics} \label{sec-maybe} -To begin we provide an overview of the \texttt{maybe} statements semantics, -describing how it structures uncertainty in variable values and runtime -execution. We refer to each of the values or code paths a \texttt{maybe} -statement can choose from as an \textit{alternative}. +To begin we provide an overview of the \texttt{maybe} statement's semantics +describing how it allows developers to structure uncertainty. We refer to +each of the values or code paths a \texttt{maybe} statement can choose from +as an \textit{alternative}. \begin{figure}[t] \begin{minted}[fontsize=\footnotesize]{java} @@ -15,7 +15,6 @@ String policy = maybe "auto", "quality", "perf"; // Function alternatives @maybe int myFunction(int a) { /* First alternative */ } - @maybe int myFunction(int a) { /* Second alternative */ } @@ -46,74 +45,34 @@ maybe { Variables can be used to represent uncertainty. Examples include an integer storing how often a timer should trigger communication with a remote server, or a string containing the name of a policy used to coordinate multiple code -blocks through the app. Figure~\ref{fig-maybeexamples} shows examples of an -integer that can take on values between 1 and 16, and a string that be set to -either ``auto'', ``quality'', or ``perf''. +blocks throughout the app. Figure~\ref{fig-maybeexamples} shows examples of +an integer that can take on values between 1 and 16, and a string that be set +to either ``auto'', ``quality'', or ``perf''. \subsection{Controlling Code Flow} -Code flow can also represent uncertainty, and we expect this to be the most -common use of \texttt{maybe} statements. Examples include evaluating -multiple algorithms to compute the same result or code paths that represent -different energy-performance or energy-quality tradeoffs. +Code flow can also represent uncertainty. Examples include using multiple +algorithms to compute the same result or multiple code paths representing +different tradeoffs between performance, energy, and quality. Figure~\ref{fig-example-maybe} shows the \texttt{maybe} statement in its simplest form controlling execution of multiple code blocks. If multiple alternatives are specified, the system chooses one to execute; if only one alternative is specified, the system chooses whether or not to execute it. -Single-alternative \texttt{maybe} statements can encapsulate -or reorganize logic that does not affect correctness, but may (or -may not) improve performance if executed. +Single-alternative \texttt{maybe} statements can encapsulate or reorganize +logic that does not affect correctness, but may (or may not) produce some +desirable objective. Figure~\ref{fig-maybeexamples} shows several extensions of the \texttt{maybe} statement providing syntactic sugar. \texttt{maybe} function annotations allow uncertainty to be expressed at the function level, with the alternatives consisting of multiple function definitions with identical -signatures. Finally, \texttt{maybe} blocks that require special performance -evaluation logic can include that in the form of a \texttt{evaluate} -statement following the main \texttt{maybe} block, as shown in the final -example. \texttt{evaluate} blocks provide app-specific \textit{a posteriori} -logic to evaluate the selected alternative. The \texttt{evaluate} block must -return a single JSON object that includes two components: A score, with -smaller being interpreted as better; and a boolean indicating whether the -system should use the same branch next time, or if it is free to try another. -Note that hints and dedicated evaluation logic can also be applied to -variable and function-level \texttt{maybe} statements through annotations. We -will return to \textit{a posteriori} evaluation in -Section~\ref{sec-certainty}. - -\subsection{Discussion} - -We have yet to determine how well programmers will be able to use the -\texttt{maybe} statement. Structurally \texttt{maybe} statements are similar -to the common \texttt{if-else} construct in that at the bottom of the -statement the programmer may not be sure which block was executed. However, -while with \texttt{if-else} blocks it can be determined which block was -executed by examining the branch conditions, \texttt{maybe} statements -require developers record which alternative was executed if downstream code -depends on the decision. To coordinate the adaptation of multiple code paths -a single \texttt{maybe} variable can be used to control multiple -\texttt{if-else} statements. - -While \texttt{maybe} is a powerful programming tool, it is important that it -be used sparingly. In the case where dependencies exist between the -alternatives expressed by separate \texttt{maybe} statements, each -\texttt{maybe} may expand the overall configuration space exponentially which -makes the resulting process of determining the best alternative much more -complicated. Compile-time analysis may be required to ensure that downstream -optimization remains feasible. - -The most important guideline for when \textit{not} to use \texttt{maybe} -usage is when a decision is not app-specific, and should instead be -refactored into a library or service. As an example, \texttt{maybe} should -not be used to decide which networking interface to use to send a packet. -To the degree that -one app shares the same objectives as others---say, to minimize latency---a -service should be performing adaptation on behalf of all apps using the -merged network interfaces, possibly through the use of its own internal -\texttt{maybe} statements. - -Finally, note that structured uncertainty is not randomness. The -\texttt{maybe} statement indicates that during any given execution one -alternative may be better than the others---even if the developer or system -are not sure which alternative to use. - +signatures. Finally, \texttt{maybe} blocks that require custom evaluation +logic can include an \texttt{evaluate} statement following the main +\texttt{maybe} block, as shown in the final example. \texttt{evaluate} blocks +provide app-specific \textit{a posteriori} logic to evaluate the selected +alternative. The \texttt{evaluate} block must return a single JSON object +with two components: (1) a positive integer \texttt{score}, with smaller +being better; (2) and a boolean \texttt{repeat} indicating whether the system +must use the same alternative next time. Hints and custom evaluation logic +can also be applied to other types of \texttt{maybe} statements through +annotations. diff --git a/paper.tex b/paper.tex index 962ec87..9803c4d 100644 --- a/paper.tex +++ b/paper.tex @@ -76,6 +76,7 @@ Jinghao Shi, Guru Prasad Srinivasa, and Lukasz Ziarek} \input{maybe.tex} \input{certainty.tex} \input{usage.tex} +\input{discussion.tex} \input{related.tex} \input{conclusion.tex} -- libgit2 0.22.2