Working.

Geoffrey Challen
1 parent 6372e677
Showing 6 changed files with 173 additions and 146 deletions
.ispell_english
certainty.tex
discussion.tex
introduction.tex
maybe.tex
paper.tex
@@ -2,4 +2,11 @@ Android
 APIs
 app
 apps
+Endgames
+JSON
+outsourced
+pre
+runtime's
+smartphone
+smartphones
 Wifi
@@ -2,85 +2,88 @@
 \label{sec-certainty}
 While \texttt{maybe} allows programmers to specify multiple alternatives,
-ultimately only one alternative may be used. Either a single, globally best
-alternative must be chosen, or a deterministic decision procedure must be
-selected. Before discussing options for \textit{adapting} an app to its
-runtime environment, we first explain our runtime's support for
+ultimately at runtime only one alternative can be executed. Either a single,
+globally-optimal alternative must be determined, or a deterministic decision
+procedure must be developed. Before discussing options for \textit{adapting}
+an app to its runtime environment, we first explain our runtime's support for
 \texttt{maybe} alternatives, including \textit{a posteriori} evaluation and
-data-collection. Then, we discuss how \texttt{maybe} testing enables a
+data collection. Then, we discuss how \texttt{maybe} testing enables a
 variety of different adaptation patterns.
-\subsection{Evaluating Alternatives} The optional \texttt{evaluate} block of
-a \texttt{maybe} statement allows programmers to provide app-specific
-\textit{a posteriori} evaluation logic. However, in many cases, we expect
-that \texttt{maybe} statements will be used to achieve common objectives such
-as improving performance or saving energy. To steamline application
-development, our current system assesses \texttt{maybe} statements without a
-\texttt{better} block in terms of both energy and performance. In cases where
-one alternative is a clear winner in terms of both energy and performance,
-that alternative will be used---although note that this selection may still
-need to be time-varying since it may depend on exogenous factors.
-
-Cases where \texttt{maybe} blocks provide an energy-performance tradeoff
-require more investigation to handle. We are exploring several options,
-including collapsing both metrics into a single score by computing the
-energy-delay product of each alternative, or allowing users to set a single
-per-app preference indicating whether energy or performance should determine
-\texttt{maybe} alternative selection.
-
-When \texttt{maybe} statements include a \texttt{better} block, their custom
-evaluation logic is expected to merge any relevant factors into a single
-score, although \texttt{better} blocks can still record other relevant
-information to aid in understanding app behavior. Specifically,
-\texttt{better} blocks return a JSON object where the ``score'' key is used
-as the score but all other keys are recorded for later analysis. Our standard
-energy and performance \texttt{better} block records energy and performance
+\subsection{Evaluating Alternatives}
+
+The optional \texttt{evaluate} block of a \texttt{maybe} statement allows
+programmers to provide app-specific \textit{a posteriori} evaluation logic.
+However, in many cases, we expect that \texttt{maybe} statements will be used
+to achieve common objectives such as improving performance or saving energy.
+To streamline application development, our current system evaluates
+\texttt{maybe} statements without a \texttt{evaluate} block by measuring both
+energy and performance. In cases where one alternative optimizes both, that
+alternative will be used---although the decision may still be time-varying
+due to dependence on time-varying factors such as network availability.
+
+\sloppypar{Cases where \texttt{maybe} statements provide an
+energy-performance tradeoff require more investigation. We are exploring
+several options, including collapsing both metrics into a single score by
+computing the energy-delay product (EDP) of each alternative, or allowing
+users to set a per-app preference indicating whether energy or performance
+should determine \texttt{maybe} alternative selection.}
+
+When \texttt{maybe} statements include a \texttt{evaluate} block, their
+custom evaluation logic must merge any relevant factors into a single score,
+although \texttt{evaluate} blocks can still record other relevant information
+to aid in understanding app behavior. Specifically, while the \texttt{score}
+value is used to evaluate the alternative, the entire JSON object returned by
+the \texttt{evaluate} block is recorded for later analysis. Our standard
+energy and performance \texttt{evaluate} block records energy and performance
 separately in the JSON object but combines them as required into a single
-score. Note that because the \texttt{better} block delivers information to
+score. Note that because the \texttt{evaluate} block delivers information to
 the developer through the \texttt{maybe} service, as described below,
 \texttt{maybe} statements can be connected with end-to-end app performance
-metrics that would normally not be visible on the device. \texttt{better}
-blocks may also want to query the user directly, and we are considering ways
-to adjust their semantics to make this possible.
-
-In some cases we expect that \texttt{better} blocks may need to know which
-alternative was executed in order to determine the score---for example, if
-the two alternatives produce different quality output while trading off
-performance or energy consumption. The simplest solution is to have each
-alternative set a variable indicating that it was used, but we believe that
-eventually including some form of labeling syntax for \texttt{maybe}
-alternatives may be beneficial.
-
-As a side note, if a \texttt{maybe} alternative encounters an error, the
-system will automatically both retry another alternative and give the
-error-generating alternative the worst possible score. In this case any
-custom evaluation logic will not be executed for the failing alternative.
-While \texttt{maybe} is not designed as a way to avoid errors, the existence
-of other alternatives provides our system with a way to work around 
-failures sometimes caused by environmental factors.
-
-A final question concerns when a \texttt{maybe} alternative should be assessed.
-In some
-cases it may be appropriate for post-mortem evaluation to happen immediately after execution.
-In others, it may be necessary to execute the same alternative over a period
-of time to perform a fair comparison. As described previously, 
-\texttt{better} blocks can indicate explicitly whether or not to continue
-evaluating the alternative, and we are determining how to make a similar
-choice available to \texttt{maybe} statements that do not use customized
-evaluation logic. In both cases, however, the \texttt{maybe} system allows
-developers continuous per-statement control over alternative evaluation and
-selection as described in more detail later in this section.
+metrics that would normally not be visible on the device. \texttt{evaluate}
+blocks may also want to query the user directly, and we are exploring ways to
+make this possible.
+
+In some cases we expect that \texttt{evaluate} blocks may need to know which
+alternative was executed to determine the score---for example, if the two
+alternatives produce different quality output. The simplest solution is to
+have each alternative set a variable indicating that it was used, but we
+believe that eventually including some form of labeling syntax for
+\texttt{maybe} alternatives may be beneficial.
+
+If a \texttt{maybe} alternative throws an error, the system will bypass the
+\texttt{evaluate} block and give it the worst possible score. By integrating
+a form of record-and-replay~\cite{FIXME}, it may also be possible to roll
+back the failed alternatives changes and retry another alternative.
+\texttt{maybe} is intended to enable adaptation, not avoid errors, but the
+existence of other alternatives provides our system with a way to work around
+failures caused by uncertainty. Resistance to errors may also encourage
+developers to use \texttt{maybe} statements to prototype alternatives to
+existing well-tested code.
+
+A final question concerns when a \texttt{maybe} alternative should be
+assessed. In some cases it may be appropriate for \textit{a posteriori}
+evaluation to happen immediately after execution. In others, it may be
+necessary to execute the same alternative over a period of time to perform a
+fair comparison. As described previously, \texttt{evaluate} blocks can
+indicate explicitly whether or not to continue evaluating the alternative,
+and we are determining how to make a similar choice available to
+\texttt{maybe} statements that do not use customized evaluation logic. In
+addition, \texttt{evaluate} blocks can store state across multiple
+alternative executions allowing them to evaluate not only micro- but also
+macro-level decisions. In both cases, however, the \texttt{maybe} system
+allows developers continuous per-statement control over alternative
+evaluation and selection as described in more detail later in this section.
 \subsection{\texttt{\large maybe} Alternative Testing}
-We next describe the pre- and post-deployment testing that helps developers to
-design an \textit{adaptation} policy, a strategy for ultimately selecting 
-between alternatives. While the \texttt{maybe} system
-automates many of the tedious tasks normally associated with large-scale
-testing, we still provide ways for the developer to guide and even override
-any step in the process.
+We next describe the pre- and post-deployment testing that helps developers
+to design an \textit{adaptation} policy, a strategy for ultimately selecting
+between alternatives. While the \texttt{maybe} system automates many of the
+tedious tasks normally associated with large-scale testing, we still provide
+ways for the developer to guide and control any step in the process.
-\subsubsection{Runtime Control}
+\subsubsection{Runtime control}
 To begin, we briefly outline how our Android prototype system implements the
 \texttt{maybe} construct. We (1) rewrite each \texttt{maybe} code block to an
@@ -108,7 +111,7 @@ so by periodically resetting the value using another \texttt{maybe}
 statement. This ensures that \texttt{maybe} variables only change when the
 developer expects them to.
-\subsubsection{Simulation or Emulation}
+\subsubsection{Simulation or emulation}
 Pre-deployment simulation or emulation may provide a way to efficiently
 assess \texttt{maybe} blocks without involving users. Building simulation
@@ -127,7 +130,7 @@ choices of which \texttt{maybe} statements may have a significant impact on
 performance and should be assessed first. Other \texttt{maybe} statements
 can be assessed later or eliminated altogether.
-\subsubsection{Split Testing}
+\subsubsection{Split testing}
 Eventually code containing a number of \texttt{maybe} statements will be
 deployed on thousands or millions of devices. At this point, large-scale
@@ -150,10 +153,10 @@ set, the \texttt{maybe} system records:
 \item what \texttt{maybe} was reached;
 \item what alternative was used and why. This includes all environmental
-features used to make the selection, as well as any other available 
+features used to make the decision, as well as any other available 
 provenance information;
-\item what \texttt{better} block evaluated the alternative, and the entire
+\item what \texttt{evaluate} block evaluated the alternative, and the entire
 JSON object it returned, including the score;
 \item and a variety of other environmental and configuration parameters 
@@ -167,7 +170,7 @@ This dataset is periodically uploaded to the \texttt{maybe} server by the
 \texttt{maybe} service and used to drive the adaptation approaches discussed
 next.
-\subsubsection{Simultaneous Split Testing}
+\subsubsection{Simultaneous split testing}
 While large-scale split testing is intended to provide good coverage over all
 possible sources of uncertainty we have discussed, it still normally requires
@@ -193,7 +196,7 @@ factors and uncertainties. We continue by discussing how the dataset
 generated by post-deployment testing can be consumed to determine how to
 correctly choose \texttt{maybe} alternatives at runtime.
-\subsubsection{Simple Cases}
+\subsubsection{Simple cases}
 In the simplest case, testing may reveal that a single alternative performs
 the best on all devices, for all users, at all times. In this situation, the
@@ -202,8 +205,8 @@ testing of that alternative and even automatically rewrite that portion of
 code to remove the \texttt{maybe} statement. However, it is also possible
 that the situation may change in the future when a new device, or Android
 version, or battery technology is introduced, and so the programmer may also
-choose to preserve the alternatives to enable later retesting and possible
-adaptation.
+choose to preserve the alternatives to enable continuous adaptation as
+described in Section~\ref{subsec-continuous}.
 The slightly more complicated case is when testing reveals that alternatives
 provide stable tradeoffs between energy and performance---one block always
@@ -215,7 +218,7 @@ each user. However, the stability of the alternatives&#39; outcomes means that
 once an energy or performance policy decision has been made, the choice of
 alternative has also been made.
-\subsubsection{Static Adaptation}
+\subsubsection{Static adaptation}
 In the more complicated cases, testing reveals that the choice of alternative
 depends on some subset of the factors driving uncertainty in mobile systems
@@ -233,7 +236,7 @@ possible that the correct alternative can be determined through some
 clustering based on these features, and once determined will remain the best
 choice over long time intervals.
-\subsubsection{Dynamic Adaptation}
+\subsubsection{Dynamic adaptation}
 If the choice of alternative depends on dynamic factors such as the accuracy
 of location services, the amount of energy left in the battery, or the type
@@ -242,7 +245,7 @@ single alternative can be chosen even for a single user. Instead, the
 \texttt{maybe} system allows developers to evaluate one or more strategies to
 drive the runtime alternative selection process.
-Note that \texttt{better} blocks are \textit{not} intended to accomplish this
+Note that \texttt{evaluate} blocks are \textit{not} intended to accomplish this
 kind of adaptation. First, they run after the \texttt{maybe} block has been
 executed, not before. Second, a single strategy defeats the flexibility
 inherent to the \texttt{maybe} approach and would devolve into the same
@@ -274,7 +277,7 @@ that time, they can connect that \texttt{maybe} statement to a sophisticated
 machine learning algorithm written by an expert in energy adaptation, instead
 of being forced to implement their own simplistic approach.
-\subsubsection{Manual Adaptation}
+\subsubsection{Manual adaptation}
 In some cases even our best efforts to automatically adapt may fail, and it
 may be impossible to predict which alternative is best for a particular user
@@ -285,3 +288,17 @@ are significant then the \texttt{maybe} alternatives may need to be exposed
 to the user through a settings menu. Fortunately, information obtained
 through testing can still be presented to the user to guide their decision.
 Note that this requires labeling alternatives in a human-readable way.
+
+\subsection{Continuous Adaptation}
+\label{subsec-continuous}
+
+Finally, even once a decision process for a particular \texttt{maybe}
+alternative has been developed, it should be periodically revisited as users,
+devices, networks, batteries, and other factors affecting mobile apps
+continue to change. To enable continuous adaptation, developers can configure
+\texttt{maybe} statements to continue to periodically experiment with
+alternatives other than the one selected by the alternative testing process.
+Changes in alternative performance relative to the expectations established
+during the last round of alternative testing may trigger a large-scale
+reexamination of that \texttt{maybe} statement using the same process
+described above.
+\section{Discussion}
+\label{sec-discussion}
+
+We have yet to determine how well programmers will be able to use the
+\texttt{maybe} statement. Structurally \texttt{maybe} statements are similar
+to the ubiquitous \texttt{if-else} construct in that at the bottom of the
+statement the programmer may not be sure which block was executed. However,
+while with \texttt{if-else} blocks it can be determined which block was
+executed by examining the branch conditions, \texttt{maybe} statements
+require developers to record which alternative was executed if downstream
+code depends on the decision. To coordinate the adaptation of multiple code
+paths a single \texttt{maybe} variable---such as the policy string in
+Figure~\ref{fig-maybeexamples}---can be used to control multiple
+\texttt{if-else} statements.
+
+While \texttt{maybe} is a powerful programming tool, it is important that it
+be used sparingly. If objective dependencies exist between \texttt{maybe}
+statements, the overall configuration space may expand exponentially,
+complicating the process of determining the best alternatives described in
+Section~\ref{sec-certainty}. Compile-time analysis may be required to detect
+dependencies between \texttt{maybe} statements and ensure that downstream
+optimization remains feasible.
+
+One important place where \texttt{maybe} should not be used is when
+adaptation is not app-specific and can be refactored into a library serving
+multiple apps. As an example, an app should not use \texttt{maybe} to decide
+which network interface to use when attempting to achieve a common objective,
+such as maximizing throughput. This choice would be better refactored into a
+dedicated library, which might use its own internal \texttt{maybe}
+statements. Not only is the resulting codebase smaller, but the total number
+of \texttt{maybe} statements that the system has to determine how to handle
+is reduced.
+
+However, the \texttt{maybe} statement represents a fundamentally different
+approach from previously attempts to enable runtime adaptation that relied on
+libraries. First, libraries
+
+Finally, note that structured uncertainty is not randomness. The
+\texttt{maybe} statement indicates that during any given execution one
+alternative may be better than the others---even if the developer or system
+are not sure which alternative to use.
@@ -88,7 +88,9 @@ maybe {
 \vspace*{-0.1in}
 \caption{\textbf{Example \texttt{maybe} Statement.} The programmer provides
-multiple equivalent alternatives.}
+multiple equivalent alternatives. The system determines how to choose between
+them at runtime.}
+
 \label{fig-example-maybe}
 \vspace*{-0.1in}
 \section{\texttt{\large maybe} Statement Semantics}
 \label{sec-maybe}
-To begin we provide an overview of the \texttt{maybe} statements semantics,
-describing how it structures uncertainty in variable values and runtime
-execution. We refer to each of the values or code paths a \texttt{maybe}
-statement can choose from as an \textit{alternative}.
+To begin we provide an overview of the \texttt{maybe} statement's semantics
+describing how it allows developers to structure uncertainty. We refer to
+each of the values or code paths a \texttt{maybe} statement can choose from
+as an \textit{alternative}.
 \begin{figure}[t]
 \begin{minted}[fontsize=\footnotesize]{java}
@@ -15,7 +15,6 @@ String policy = maybe &quot;auto&quot;, &quot;quality&quot;, &quot;perf&quot;;
 // Function alternatives
 @maybe
 int myFunction(int a) { /* First alternative */ }
-
 @maybe
 int myFunction(int a) { /* Second alternative */ }
@@ -46,74 +45,34 @@ maybe {
 Variables can be used to represent uncertainty. Examples include an integer
 storing how often a timer should trigger communication with a remote server,
 or a string containing the name of a policy used to coordinate multiple code
-blocks through the app. Figure~\ref{fig-maybeexamples} shows examples of an
-integer that can take on values between 1 and 16, and a string that be set to
-either ``auto'', ``quality'', or ``perf''.
+blocks throughout the app. Figure~\ref{fig-maybeexamples} shows examples of
+an integer that can take on values between 1 and 16, and a string that be set
+to either ``auto'', ``quality'', or ``perf''.
 \subsection{Controlling Code Flow}
-Code flow can also represent uncertainty, and we expect this to be the most
-common use of \texttt{maybe} statements. Examples include evaluating
-multiple algorithms to compute the same result or code paths that represent
-different energy-performance or energy-quality tradeoffs.
+Code flow can also represent uncertainty. Examples include using multiple
+algorithms to compute the same result or multiple code paths representing
+different tradeoffs between performance, energy, and quality.
 Figure~\ref{fig-example-maybe} shows the \texttt{maybe} statement in its
 simplest form controlling execution of multiple code blocks. If multiple
 alternatives are specified, the system chooses one to execute; if only one
 alternative is specified, the system chooses whether or not to execute it.
-Single-alternative \texttt{maybe} statements can encapsulate 
-or reorganize logic that does not affect correctness, but may (or
-may not) improve performance if executed.
+Single-alternative \texttt{maybe} statements can encapsulate or reorganize
+logic that does not affect correctness, but may (or may not) produce some
+desirable objective.
 Figure~\ref{fig-maybeexamples} shows several extensions of the \texttt{maybe}
 statement providing syntactic sugar. \texttt{maybe} function annotations
 allow uncertainty to be expressed at the function level, with the
 alternatives consisting of multiple function definitions with identical
-signatures. Finally, \texttt{maybe} blocks that require special performance
-evaluation logic can include that in the form of a \texttt{evaluate}
-statement following the main \texttt{maybe} block, as shown in the final
-example. \texttt{evaluate} blocks provide app-specific \textit{a posteriori}
-logic to evaluate the selected alternative. The \texttt{evaluate} block must
-return a single JSON object that includes two components: A score, with
-smaller being interpreted as better; and a boolean indicating whether the
-system should use the same branch next time, or if it is free to try another.
-Note that hints and dedicated evaluation logic can also be applied to
-variable and function-level \texttt{maybe} statements through annotations. We
-will return to \textit{a posteriori} evaluation in
-Section~\ref{sec-certainty}.
-
-\subsection{Discussion}
-
-We have yet to determine how well programmers will be able to use the
-\texttt{maybe} statement. Structurally \texttt{maybe} statements are similar
-to the common \texttt{if-else} construct in that at the bottom of the
-statement the programmer may not be sure which block was executed. However,
-while with \texttt{if-else} blocks it can be determined which block was
-executed by examining the branch conditions, \texttt{maybe} statements
-require developers record which alternative was executed if downstream code
-depends on the decision. To coordinate the adaptation of multiple code paths
-a single \texttt{maybe} variable can be used to control multiple
-\texttt{if-else} statements.
-
-While \texttt{maybe} is a powerful programming tool, it is important that it
-be used sparingly. In the case where dependencies exist between the
-alternatives expressed by separate \texttt{maybe} statements, each
-\texttt{maybe} may expand the overall configuration space exponentially which
-makes the resulting process of determining the best alternative much more
-complicated. Compile-time analysis may be required to ensure that downstream
-optimization remains feasible.
-
-The most important guideline for when \textit{not} to use \texttt{maybe}
-usage is when a decision is not app-specific, and should instead be 
-refactored into a library or service. As an example, \texttt{maybe} should 
-not be used to decide which networking interface to use to send a packet. 
-To the degree that
-one app shares the same objectives as others---say, to minimize latency---a
-service should be performing adaptation on behalf of all apps using the
-merged network interfaces, possibly through the use of its own internal
-\texttt{maybe} statements.
-
-Finally, note that structured uncertainty is not randomness. The
-\texttt{maybe} statement indicates that during any given execution one
-alternative may be better than the others---even if the developer or system
-are not sure which alternative to use.
-
+signatures. Finally, \texttt{maybe} blocks that require custom evaluation
+logic can include an \texttt{evaluate} statement following the main
+\texttt{maybe} block, as shown in the final example. \texttt{evaluate} blocks
+provide app-specific \textit{a posteriori} logic to evaluate the selected
+alternative. The \texttt{evaluate} block must return a single JSON object
+with two components: (1) a positive integer \texttt{score}, with smaller
+being better; (2) and a boolean \texttt{repeat} indicating whether the system
+must use the same alternative next time. Hints and custom evaluation logic
+can also be applied to other types of \texttt{maybe} statements through
+annotations.
@@ -76,6 +76,7 @@ Jinghao Shi, Guru Prasad Srinivasa, and Lukasz Ziarek}
 \input{maybe.tex}
 \input{certainty.tex}
 \input{usage.tex}
+\input{discussion.tex}
 \input{related.tex}
 \input{conclusion.tex}