diff --git a/certainty.tex b/certainty.tex index 094a468..83d4794 100644 --- a/certainty.tex +++ b/certainty.tex @@ -17,58 +17,44 @@ variety of different adaptation patterns. The optional \texttt{evaluate} block of a \texttt{maybe} statement allows programmers to provide app-specific \textit{a posteriori} evaluation logic. However, in many cases, we expect that \texttt{maybe} statements will be used -to achieve common objectives such as improving performance or saving energy. -To streamline application development, our current system evaluates -\texttt{maybe} statements without a \texttt{evaluate} block by measuring both -energy and performance. In cases where one alternative optimizes both, that -alternative will be used---although the decision may still be time-varying -due to dependence on time-varying factors such as network availability. - -\sloppypar{Cases where \texttt{maybe} statements provide an -energy-performance tradeoff require more investigation. We are exploring -several options, including collapsing both metrics into a single score by -computing the energy-delay product (EDP) of each alternative, or allowing -users to set a per-app preference indicating whether energy or performance -should determine \texttt{maybe} alternative selection.} - -When \texttt{maybe} statements include a \texttt{evaluate} block, their -custom evaluation logic must merge any relevant factors into a single score, -although \texttt{evaluate} blocks can still record other relevant information -to aid in understanding app behavior. Specifically, while the \texttt{score} -value is used to evaluate the alternative, the entire JSON object returned by -the \texttt{evaluate} block is recorded for later analysis. Our standard -energy and performance \texttt{evaluate} block records energy and performance -separately in the JSON object but combines them as required into a single -score. Note that because the \texttt{evaluate} block delivers information to -the developer through the \texttt{maybe} service, as described below, -\texttt{maybe} statements can be connected with end-to-end app performance -metrics that would normally not be visible on the device. \texttt{evaluate} -blocks may also want to query the user directly, and we are exploring ways to -make this possible. - -In some cases we expect that \texttt{evaluate} blocks may need to know which -alternative was executed to determine the score---for example, if the two -alternatives produce different quality output. The simplest solution is to -have each alternative set a variable indicating that it was used, but we -believe that eventually including some form of labeling syntax for -\texttt{maybe} alternatives may be beneficial. +to achieve common objectives such as improving performance or saving energy. To +streamline application development, our current system evaluates \texttt{maybe} +statements without a \texttt{evaluate} block by measuring both energy and +performance. In cases where one alternative optimizes both, that alternative +will be used---although the decision may still be time-varying due to +dependence on time-varying factors such as network availability. When +alternatives produce an energy-performance tradeoff we are exploring several +options, including collapsing both metrics into a single score by computing the +energy-delay product (EDP) of each alternative, or allowing users to set a +per-app energy or performance preference. + +\texttt{evaluate} blocks can also record other information to aid adaptation. +While the \texttt{score} value is used to evaluate the alternative, the +entire JSON object returned by the \texttt{evaluate} block is delivered to +the developer for later analysis. This allows \texttt{maybe} statements to be +connected with end-to-end app performance metrics not visible on the device. +\texttt{evaluate} blocks may also want to query the user directly, and we are +exploring ways to make this possible. + +We expect that some \texttt{evaluate} blocks may need to know which +alternative was executed to compute a score---for example, if the two +alternatives produce different quality output. We are exploring the use of +automatically-generated labels to aid this process. If a \texttt{maybe} alternative throws an error, the system will bypass the \texttt{evaluate} block and give it the worst possible score. By integrating -a form of record-and-replay~\cite{gomez2013reran}, it may also be possible to -roll back the failed alternatives changes and retry another alternative. -\texttt{maybe} is intended to enable adaptation, not avoid errors, but the -existence of other alternatives provides our system with a way to work around -failures caused by uncertainty. Resistance to errors may also encourage -developers to use \texttt{maybe} statements to prototype alternatives to -existing well-tested code. +a form of record-and-replay~\cite{gomez2013reran}, it may be possible to roll +back the failed alternative and retry another. \texttt{maybe} is intended to +enable adaptation, not avoid errors, but the existence of other alternatives +provides a way to work around failures caused by uncertainty. Fault tolerance +may also encourage developers to use \texttt{maybe} statements to prototype +alternatives to existing well-tested code. A final question concerns when a \texttt{maybe} alternative should be -evaluated. In some cases it may be appropriate for \textit{a posteriori} -evaluation to happen immediately after execution. In others, it may be -necessary to continue executing the same alternative over a period of time to -perform a fair comparison. As described previously, \texttt{evaluate} blocks -can indicate explicitly whether or not to continue evaluating the +evaluated. Some alternatives may require evaluation immediately after +execution. Others may require repeated execution over a longer period of time +to perform a fair comparison. As described previously, \texttt{evaluate} +blocks can indicate explicitly whether or not to continue evaluating the alternative, and we are determining how to make a similar choice available to \texttt{maybe} statements without \texttt{evaluate} blocks. In addition, \texttt{evaluate} blocks can store state across multiple alternative diff --git a/conclusion.tex b/conclusion.tex index edd3f4d..af030e2 100644 --- a/conclusion.tex +++ b/conclusion.tex @@ -1,4 +1,3 @@ -\vspace*{-0.05in} \section{Conclusion} \label{sec-conclusion} @@ -8,7 +7,6 @@ development time and for that uncertainty to be resolved through later testing and adaptation. We are in the process of building a prototype of the \texttt{maybe} system for Android smartphones. -\vspace*{-0.05in} \section*{Acknowledgments} Students and faculty working on the \texttt{maybe} project are supported by @@ -19,5 +17,3 @@ and \href{http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=1423215}{1423215}. The \texttt{maybe} team thanks the anonymous reviewers and our shepherd, Mahadev Satyanarayanan, for their feedback. - -\vspace*{-0.05in} diff --git a/discussion.tex b/discussion.tex index 98bd89e..06ebfe0 100644 --- a/discussion.tex +++ b/discussion.tex @@ -5,44 +5,33 @@ We have yet to determine how natural programmers will find the \texttt{maybe} statement. Encouragingly, \texttt{maybe} statements are similar to the -ubiquitous \texttt{if-else} statement, and in many cases can be used to -directly replace \texttt{if-else} statements that attempt runtime adaptation. -To coordinate the adaptation of multiple code paths a single \texttt{maybe} -variable---such as the policy string in Figure~\ref{fig-maybeexamples}---can -be used to control multiple \texttt{if-else} statements. +ubiquitous \texttt{if-else} statement, and in many cases can directly replace +\texttt{if-else} statements that attempt runtime adaptation. To coordinate +the adaptation of multiple code paths a single \texttt{maybe} variable can be +used to control multiple \texttt{if-else} statements. -While \texttt{maybe} is a powerful programming tool, it is important that it -be used sparingly. If objective dependencies exist between \texttt{maybe} -statements, the overall configuration space may expand exponentially, -complicating post-deployment adaptation process. Compile-time analysis may be -required to detect dependencies between \texttt{maybe} statements and -encourage programmers to limit their use of \texttt{maybe} if to ensure that -downstream optimization remains feasible. +Overuse of the \texttt{maybe} statement may cause problems. If dependencies +exist between \texttt{maybe} statements, the overall configuration space may +expand exponentially, complicating post-deployment adaptation. Compile-time +analysis may be required to detect dependencies between \texttt{maybe} +statements and encourage programmers to limit their use of \texttt{maybe} to +ensure that downstream optimization remains feasible. -\texttt{maybe} statements should not be used when adaptation is not -app-specific and can be refactored into a library. As an example, an app -should not use \texttt{maybe} to decide which network interface to use when -attempting to achieve a common objective, such as maximizing throughput. This -choice would be better refactored into a dedicated library, which might use -its own internal \texttt{maybe} statements. Not only is the resulting -codebase smaller, but the total number of \texttt{maybe} statements that the -system has to test is reduced. +\texttt{maybe} statements should not be used when adaptation can be +refactored into a library. As an example, an app should not use +\texttt{maybe} to decide which network interface to use when attempting to +achieve a common objective, such as maximizing throughput. This adaptation +should be refactored into a dedicated library, which might use its own +\texttt{maybe} statements. Not only is the resulting codebase smaller, but +the total number of \texttt{maybe} statements to test is reduced. However, the \texttt{maybe} statement represents a fundamentally different -approach from attempts to enable runtime adaptation that rely on libraries. -As a code organization strategy, the benefits that libraries provide in terms -of reduced duplication and more powerful interfaces are orthogonal and -complementary to the \texttt{maybe} statement. However, a large amount of -code including adaptation logic remains app specific and cannot be refactored -into a library. - -More importantly, library development still requires development-time -certainty. Despite the fact that library developers are more likely to be -experts at the type of adaptation the library performs, we still believe that -even the most skilled programmers are not capable of anticipating all -possible sources of uncertainty and will benefit from being able to express -structured uncertainty. \texttt{maybe} allows all developers---including both -app and library writers---to shed the burden of producing a single certain -approach and instead write uncertain code containing the flexibility needed -required to enable powerful data-driven approaches to post-deployment -adaptation. +approach to runtime adaptation than systems that rely on libraries because +library development still requires development-time certainty. While library +developers are more likely to be experts at the type of adaptation their +library performs, we still believe that even the most skilled programmers +will benefit from being able to express structured uncertainty. +\texttt{maybe} allows all developers---including both app and library +writers---to shed the burden of producing a single certain approach and +instead write uncertain code containing the flexibility required to enable +powerful data-driven approaches to post-deployment adaptation. diff --git a/related.tex b/related.tex index 75d03c5..015a09c 100644 --- a/related.tex +++ b/related.tex @@ -7,21 +7,20 @@ programming with uncertain data, rather than the runtime adaptation enabled by \texttt{maybe}. Aspect oriented programming (AOP)~\cite{aop} aims to increase modularity -through the separation of cross-cutting concerns. The programmer can express -cross-cutting concerns in stand alone modules, or aspects. Aspects are -composed of advice, which specifies a computation to be performed as well as -points in the program at which that computation should be performed. -Fundamentally, the goals of AOP and the \texttt{maybe} statement differ, with -AOP focusing on modularity and \texttt{maybe} focused on increasing runtime -flexibility in the face of uncertainty. +through the separation of cross-cutting concerns. The programmer expresses +cross-cutting concerns in stand alone modules, or aspects, which specify a +computation to be performed as well as points in the program at which that +computation should be performed. Fundamentally, the goals of AOP and the +\texttt{maybe} statement differ, with AOP focusing on modularity and +\texttt{maybe} focused on enabling adaptation by expressing uncertainty. -\texttt{maybe} shares similaries with language-based approaches to managing +\texttt{maybe} shares similaries with language-based approaches to adapting energy consumption such as Eon~\cite{sensys07-eon} and Levels~\cite{sensys07-levels}. However, these approaches still require -programmers to express certainty by associating code with particular energy -states, rather than allowing the \texttt{maybe} system to determine which -energy states are appropriate. \texttt{maybe} can also enable adaptation -driven by goals other than energy management. +programmers to express certainty by associating code with energy states, +rather than allowing the \texttt{maybe} system to determine which energy +states are appropriate. \texttt{maybe} can also enable adaptation driven by +goals other than energy management. Attempts to enable more adaptive mobile systems date back to systems such as Odyssey~\cite{odyssey-sosp97}. However, a taxonomy of approaches to enabling @@ -30,3 +29,16 @@ the focus of early efforts on incorporating adaptation into libraries that could be used by multiple apps. As we have pointed out previously, while adaptation libraries are useful, \texttt{maybe} statements can make them more powerful by allowing programmers to express uncertainty. + +Recent approaches that allow mobile devices to effectively offload +computation by automating client-cloud partitioning are also related to the +\texttt{maybe} statement. Systems such as Tactics~\cite{tactics-mobisys03} +and MAUI~\cite{FIXME} used a variety of approaches to enabling this form of +adaptation but are narrowly-focused on harnessing opportunities for remote +execution. While \texttt{maybe} statements can enable simple decisions about +whether to offload portions of an app, the \texttt{maybe} approach is not yet +rich enough to consider how much of the execution should be performed +remotely or deal with server failures. At present \texttt{maybe} focuses on +single-device adaptation, but we are interested in exploring the ability to +use uncertainty to distribute computation between multiple devices as future +work.