camera ready version #2

Anudipa Maiti
1 parent 787d6a35
Showing 9 changed files with 207 additions and 200 deletions
conclusion.tex~
figures/tables/tableALL.tex
figures/tables/tableALL.tex~
figures/tables/tableRATE.tex
figures/tables/tableRATE.tex~
introduction.tex
metric.tex
results.tex
results.tex~
-\section{Conclusions}
-\label{sec-conclusion}
-
-To conclude, we have argued that our inability to estimate app value is a
-critical weakness that is threatening our successes at accurately estimating
-and attributing energy consumption. We have motivated the need for a value
-measure by describing the multiple ways in which it would aid in the
-management of energy and other resources on battery-powered smartphones.
-Using an energy consumption dataset collected on \PhoneLab{} we have explored
-separately several potential inputs to a value measure and determined how
-they weight energy consumption. And finally, we have presented results from a
-failed effort to formulate an effective value measure. While this first
-attempt was unsuccessful, we hope to engage the mobile systems community in
-this effort so that more sophisticated and successful value measures can be
-developed.
-
-\section*{Acknowledgments}
-
-Students and faculty working on estimating app value are supported by NSF
-awards
-\href{http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=1205656}{1205656}
-and
-\href{http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=1423215}{1423215}.
-The authors thank the anonymous reviewers for their feedback.
 \begin{table*}[t]
 \begin{subtable}[t]{0.5\textwidth}
 {\small
-\begin{tabularx}{\columnwidth}{rXr}
+\begin{tabularx}{\columnwidth}{rrr}
 \multicolumn{1}{c}{\normalsize{\textbf{Rank}}} & 
 \multicolumn{1}{c}{\normalsize{\textbf{App}}} & 
 \multicolumn{1}{c}{\normalsize{\textbf{Energy (As)}}} \\
@@ -34,10 +34,10 @@
 \end{subtable}%
 \begin{subtable}[t]{0.5\textwidth}
 {\small
-\begin{tabularx}{\columnwidth}{Xrr}
+\begin{tabularx}{\columnwidth}{rrc}
 \multicolumn{1}{c}{\normalsize{\textbf{Rank}}} & 
-\multicolumn{1}{c}{\normalsize{\textbf{Consumption Rate (A)}}} & 
-\multicolumn{1}{c}{\normalsize{\textbf{Value}}} \\
+\multicolumn{1}{c}{\normalsize{\textbf{App}}} & 
+\multicolumn{1}{c}{\normalsize{\textbf{Consumption Rate (A)}}} \\
 \toprule
 \num{1} & \texttt{Facebook Messenger} & 0.774 \\
 \num{2} & \texttt{Google+} & 0.614 \\
@@ -70,7 +70,7 @@
 \end{subtable}\\[0.1in]
 \begin{subtable}[t]{0.5\textwidth}
 {\small
-\begin{tabularx}{\columnwidth}{Xrr}
+\begin{tabularx}{\columnwidth}{@{\extracolsep{\fill}}rrrr}
 \multicolumn{1}{c}{\normalsize{\textbf{Rank}}} & 
 \multicolumn{1}{c}{\normalsize{\textbf{App Name}}} & 
 \multicolumn{1}{c}{\normalsize{\textbf{Efficiency}}} \\
@@ -106,7 +106,7 @@
 \end{subtable}%
 \begin{subtable}[t]{0.5\textwidth}
 {\small
-\begin{tabularx}{\columnwidth}{Xrr}
+\begin{tabularx}{\columnwidth}{@{\extracolsep{\fill}}rrrr}
 \multicolumn{1}{c}{\normalsize{\textbf{Rank}}} & 
 \multicolumn{1}{c}{\normalsize{\textbf{App Name}}} & 
 \multicolumn{1}{c}{\normalsize{\textbf{Value}}} \\
+\begin{table*}[t]
+\begin{subtable}[t]{0.5\textwidth}
+{\small
+\begin{tabularx}{\columnwidth}{rrr}
+\multicolumn{1}{c}{\normalsize{\textbf{Rank}}} & 
+\multicolumn{1}{c}{\normalsize{\textbf{App}}} & 
+\multicolumn{1}{c}{\normalsize{\textbf{Energy (As)}}} \\
+\toprule
+\num{1} & \texttt{Android Browser} & 41052.703 \\
+\num{2} & \texttt{Facebook} & 37268.388 \\
+\num{3} & \texttt{Chrome Browser} & 22719.020 \\
+\num{4} & \texttt{Android Phone} & 18122.433 \\
+\num{5} & \texttt{Gmail} & 17402.896 \\
+\num{6} & \texttt{Android Messaging} & 17342.926 \\
+\num{7} & \texttt{WhatsApp Messenger} & 16467.477 \\
+\num{8} & \texttt{Google Search} & 15370.252 \\
+\num{9} & \texttt{Candy Crush Saga} & 12767.649 \\
+\num{10} & \texttt{Android Gallery} & 11050.363 \\
+\bottomrule
+\num{10} & \texttt{Google+} & 586.586 \\
+\num{9} & \texttt{Android Calculator} & 449.474 \\
+\num{8} & \texttt{NFL Mobile} & 344.492 \\
+\num{7} & \texttt{UB Parking} & 311.766 \\
+\num{6} & \texttt{Super-Bright LED Flashlight} & 218.870 \\
+\num{5} & \texttt{Starbucks} & 174.609 \\
+\num{4} & \texttt{Google Keep} & 174.263 \\
+\num{3} & \texttt{Dropbox} & 160.939 \\
+\num{2} & \texttt{ESPN SportsCenter} & 108.965 \\
+\num{1} & \texttt{Bank of America} & 98.007 \\
+\end{tabularx}
+}
+\caption{\small \textbf{Most and Least Energy-Consuming Apps.}}
+\label{table-total}
+\end{subtable}%
+\begin{subtable}[t]{0.5\textwidth}
+{\small
+\begin{tabularx}{\columnwidth}{rrc}
+\multicolumn{1}{c}{\normalsize{\textbf{Rank}}} & 
+\multicolumn{1}{c}{\normalsize{\textbf{App}}} & 
+\multicolumn{1}{c}{\normalsize{\textbf{Consumption Rate (A)}}} \\
+\toprule
+\num{1} & \texttt{Facebook Messenger} & 0.774 \\
+\num{2} & \texttt{Google+} & 0.614 \\
+\num{3} & \texttt{Super-Bright LED Flashlight} & 0.600 \\
+\num{4} & \texttt{UB Parking} & 0.598 \\
+\num{5} & \texttt{Android Music} & 0.446 \\
+\num{6} & \texttt{Google Search} & 0.428 \\
+\num{7} & \texttt{NFL Mobile} & 0.386 \\
+\num{8} & \texttt{Pandora} & 0.326 \\
+\num{9} & \texttt{Starbucks} & 0.282 \\
+\num{10} & \texttt{Android News and Weather} & 0.254 \\
+\bottomrule
+\num{10} & \texttt{Chrome Browser} & 0.099 \\
+\num{9} & \texttt{WhatsApp Messenger} & 0.095 \\
+\num{8} & \texttt{Twitter} & 0.078 \\
+\num{7} & \texttt{Yahoo Mail} & 0.077 \\
+\num{6} & \texttt{Android Messaging} & 0.061 \\
+\num{5} & \texttt{Skype} & 0.040 \\
+\num{4} & \texttt{YouTube} & 0.036 \\
+\num{3} & \texttt{ESPN SportsCenter} & 0.021 \\
+\num{2} & \texttt{The Weather Channel} & 0.019 \\
+\num{1} & \texttt{Bank of America} & 0.011 \\
+\end{tabularx}
+}
+
+\caption{\small \textbf{Fastest and Slowest Energy-Consuming Apps.}}
+
+\label{table-rate}
+
+\end{subtable}\\[0.1in]
+\begin{subtable}[t]{0.5\textwidth}
+{\small
+\begin{tabularx}{\columnwidth}{rrr}
+\multicolumn{1}{c}{\normalsize{\textbf{Rank}}} & 
+\multicolumn{1}{c}{\normalsize{\textbf{App Name}}} & 
+\multicolumn{1}{c}{\normalsize{\textbf{Efficiency}}} \\
+\toprule
+\num{1} & \texttt{Bank of America} & 83.717 \\
+\num{2} & \texttt{The Weather Channel} & 49.861 \\
+\num{3} & \texttt{Skype} & 23.779 \\
+\num{4} & \texttt{YouTube} & 19.880 \\
+\num{5} & \texttt{Android Messaging} & 12.933 \\
+\num{6} & \texttt{Android Gallery} & 9.260 \\
+\num{7} & \texttt{Android Calculator} & 9.189 \\
+\num{8} & \texttt{Twitter} & 8.645 \\
+\num{9} & \texttt{Chrome Browser} & 8.524 \\
+\num{10} & & \\
+\bottomrule
+\num{10} & \texttt{Yahoo Mail} & 3.287 \\
+\num{9} & \texttt{ESPN SportsCenter} & 3.184 \\
+\num{8} & \texttt{Google Search} & 1.984 \\
+\num{7} & \texttt{Android Music} & 1.972 \\
+\num{6} & \texttt{Pandora} & 1.779 \\
+\num{5} & \texttt{Super-Bright LED Flashlight} & 1.667 \\
+\num{4} & \texttt{UB Parking} & 1.507 \\
+\num{3} & \texttt{NFL Mobile} & 1.437 \\
+\num{2} & \texttt{Google+} & 1.270 \\
+\num{1} & \texttt{Facebook Messenger} & 1.199 \\
+\end{tabularx}
+}
+
+\caption{\small \textbf{Apps Sorted by Foreground Energy Efficiency.}}
+
+\label{table-foreground}
+
+\end{subtable}%
+\begin{subtable}[t]{0.5\textwidth}
+{\small
+\begin{tabularx}{\columnwidth}{@{\extracolsep{\fill}}rrrr}
+\multicolumn{1}{c}{\normalsize{\textbf{Rank}}} & 
+\multicolumn{1}{c}{\normalsize{\textbf{App Name}}} & 
+\multicolumn{1}{c}{\normalsize{\textbf{Value}}} \\
+\toprule
+\num{1} & \texttt{YouTube} & 18497.052 \\
+\num{2} & \texttt{Candy Crush Saga} & 14051.369 \\
+\num{3} & \texttt{Bank of America} & 12954.196 \\
+\num{4} & \texttt{Dropbox} & 7063.746 \\
+\num{5} & \texttt{Android Messaging} & 6555.140 \\
+\num{6} & \texttt{Android Gallery} & 5773.902 \\
+\num{7} & \texttt{Twitter} & 5610.394 \\
+\num{8} & \texttt{Android Clock} & 5085.873 \\
+\num{9} & \texttt{Yahoo Mail} & 5083.615 \\
+\num{10} & & \\
+\bottomrule
+\num{10} & \texttt{NFL Mobile} & 1275.985 \\
+\num{9} & \texttt{UB Parking} & 1071.529 \\
+\num{8} & \texttt{Pandora} & 1049.971 \\
+\num{7} & \texttt{Facebook Messenger} & 1012.536 \\
+\num{6} & \texttt{Android News and Weather} & 990.386 \\
+\num{5} & \texttt{Adobe Reader} & 985.680 \\
+\num{4} & \texttt{Google+} & 898.589 \\
+\num{3} & \texttt{Android Phone} & 748.077 \\
+\num{2} & \texttt{Google Search} & 682.005 \\
+\num{1} & \texttt{The Weather Channel} & 571.405 \\
+\end{tabularx}
+}
+
+\caption{\small \textbf{Apps Sorted by Content Energy Efficiency.}}
+
+\label{table-content}
+\end{subtable}
+
+\caption{\small \textbf{Evaluating Components of a Value Measure.}
+ \PhoneLab{} data is used to weight overall app energy usage in a variety of
+different ways. Omitted results are caused by Android reporting energy
+consumption for non-apps such as the Android System.}
+
+\label{table-results}
+
+\end{table*}
 \begin{table}[t]
 {\small
-\begin{tabularx}{\columnwidth}{Xrr}
+\begin{tabularx}{\columnwidth}{rXr}
 \multicolumn{1}{c}{\normalsize{\textbf{Rank}}} & 
-\multicolumn{1}{c}{\normalsize{\textbf{Consumption Rate (A)}}} & 
-\multicolumn{1}{c}{\normalsize{\textbf{Value}}} \\
+\multicolumn{1}{c}{\normalsize{\textbf{App}}} & 
+\multicolumn{1}{c}{\normalsize{\textbf{Consumption Rate (A)}}} \\
 \toprule
 \num{1} & \texttt{Facebook Messenger} & 0.774 \\
 \num{2} & \texttt{Google+} & 0.614 \\
+\begin{table}[t]
+{\small
+\begin{tabularx}{\columnwidth}{rXr}
+\multicolumn{1}{c}{\normalsize{\textbf{Rank}}} & 
+\multicolumn{1}{c}{\normalsize{\textbf{App}}} & 
+\multicolumn{1}{c}{\normalsize{\textbf{Consumption Rate (A)}}} \\
+\toprule
+\num{1} & \texttt{Facebook Messenger} & 0.774 \\
+\num{2} & \texttt{Google+} & 0.614 \\
+\num{3} & \texttt{Super-Bright LED Flashlight} & 0.600 \\
+\num{4} & \texttt{UB Parking} & 0.598 \\
+\num{5} & \texttt{Android Music} & 0.446 \\
+\num{6} & \texttt{Google Search} & 0.428 \\
+\num{7} & \texttt{NFL Mobile} & 0.386 \\
+\num{8} & \texttt{Pandora} & 0.326 \\
+\num{9} & \texttt{Starbucks} & 0.282 \\
+\num{10} & \texttt{Android News and Weather} & 0.254 \\
+\bottomrule
+\bottomrule
+\num{9} & \texttt{WhatsApp Messenger} & 0.095 \\
+\num{8} & \texttt{Twitter} & 0.078 \\
+\num{7} & \texttt{Yahoo Mail} & 0.077 \\
+\num{6} & \texttt{Android Messaging} & 0.061 \\
+\num{5} & \texttt{Skype} & 0.040 \\
+\num{4} & \texttt{YouTube} & 0.036 \\
+\num{3} & \texttt{ESPN SportsCenter} & 0.021 \\
+\num{2} & \texttt{The Weather Channel} & 0.019 \\
+\num{1} & \texttt{Bank of America} & 0.011 \\
+\end{tabularx}
+}
+
+\caption{\small \textbf{Fastest and slowest energy-consuming apps.}}
+
+\label{table-rate}
+
+\end{table}
@@ -41,8 +41,8 @@ of apps in order to evaluate two video conferencing tools, web browsers, or
 email clients. Developers can determine whether a new feature delivers value
 more or less efficiently than the rest of their app and better understand the
 differences in energy consumption across different users. Measuring value
-allows a rigorous definition of an \textit{energy virus} as \textit{an app that
-delivers little or no value per joule}, and for systems to reward efficient
+allows a rigorous definition of an \textit{energy virus} as an app that
+delivers little or no value per joule, and for systems to reward efficient
 apps by prioritizing limited resources based on app value or energy
 efficiency. After all the progress we have made in computing the
 denominator---energy consumption---we believe that the search for the missing
@@ -33,7 +33,7 @@ with complementary value measure components that offset the differences
 between high-interaction and low-interaction apps. This approach also fails
 in the case where apps deploy confusing or unnecessary interfaces that
 require a great deal of unnecessary interaction to accomplish simple tasks.
-Clearly such apps should not be rewarded.
+Clearly, such apps should not be rewarded.
  
 \subsection{Notification Click-Through Rates}
  
@@ -61,7 +61,7 @@ not consumed did not need to be retrieved.
  
 However, in some cases apps may do an effective job at summarizing the event
 within the notification itself, providing no need for the user to bring the
-app to the foreground. Clearly such apps should not be penalized.
+app to the foreground. Clearly, such apps should not be penalized.
  
 \subsection{Content Delivery}
  
@@ -88,7 +88,8 @@ large amount of video content to their users per joule.
 presented to the user and efficiency is determined by the accelerometer rate
 and any post-processing required to produce an accurate estimate. Value is
 measured as the ability to maintain the step count and efficient pedometers
-can compute accurate values while consuming small amounts of energy.
+can achieve more accuracy in computing values per joule.
+%can compute accurate values while consuming small amounts of energy.
  
 \end{itemize}
  
@@ -25,7 +25,7 @@ package to record energy consumption at each app transition. This allows energy
 consumption by components such as the screen to be accurately attributed to
 the foreground app, a feature that Android's internal battery monitoring
 component (the Fuel Gauge) lacks. Changes were distributed to \PhoneLab{}
-participants in November, 2013, via an over-the-air (OTA) platform update.
+participants in November 2013 via an over-the-air (OTA) platform update.
 The resulting 2~month dataset of 67~GB of compressed log files represents
 \num{6806} user days during which \num{1328}~apps were started \num{277785}
 times, and used for a total of \num{15224} hours of active use by
@@ -37,17 +37,18 @@ consumed by each app. Next, we formulate a simple measure of content
 delivery by measuring usage of the screen and audio output devices and test
 it through a survey completed by 47~experiment participants. Unfortunately,
 our results are inconclusive and open to several possible interpretations
-which we discuss. We present our results in tabular format where for each measure we 
-rank 10 best performing and 10 worst performing apps in desending order.
+which we discuss. 
+We present our results in tabular format where for each measure we 
+rank 10 best performing and 10 worst performing apps in descending order.
  
-\newpage
+%\newpage
  
 \subsection{Total Energy}
  
 \input{./figures/tables/tableALL.tex}
  
 Clearly, ranking apps by total energy consumption computed by adding all 
-foreground and background energy consumptions over the entire study says 
+foreground and background energy consumption over the entire study says 
 much more about app popularity than it does about anything else.
 Table~\ref{table-total} shows the top and bottom energy-consuming apps over
 the entire study. As expected, popular apps such as the Android Browser,
-\section{Results}
-\label{sec-results}
-
-To examine the potential components of a value measure further, we utilize a
-large dataset of energy consumption measurements collected by an IRB-approved
-experiment run on the \PhoneLab{} testbed. \PhoneLab{} is a public smartphone
-platform testbed located at the University at
-Buffalo~\cite{phonelab-sensemine13}. 220~students, faculty, and staff carry
-instrumented Android Nexus~5 smartphones and receive subsidized service in
-return for willingness to participate in experiments. \PhoneLab{} provides
-access to a representative group of participants balanced between genders and
-across a wide variety of age brackets, making our results more
-representative.
-
-Understanding fine-grained energy consumption dynamics required more
-information than Android normally exposes to apps. In addition, to explore
-components of our value measure we also wanted to capture information about
-app usage---including foreground and background time and use of the display
-and audio interface---that was not possible to measure on unmodified Android
-devices. So to collect our dataset we took advantage of \PhoneLab{}'s ability
-to modify the Android platform itself. We instrumented the
-\texttt{SurfaceFlinger} and \texttt{AudioFlinger} components in the Android platform
-to record usage of the screen and audio, and altered the ActivityManagerService
-package to record energy consumption at each app transition. This allows energy
-consumption by components such as the screen to be accurately attributed to
-the foreground app, a feature that Android's internal battery monitoring
-component (the Fuel Gauge) lacks. Changes were distributed to \PhoneLab{}
-participants in November, 2013, via an over-the-air (OTA) platform update.
-The resulting 2~month dataset of 67~GB of compressed log files represents
-\num{6806} user days during which \num{1328}~apps were started \num{277785}
-times, and used for a total of \num{15224} hours of active use by
-107~\PhoneLab{} participants.
-
-Our analysis begins by investigating several components of a possible value
-measure and shows the effect of using each to weight the overall energy
-consumed by each app. Next, we formulate a simple measure of content
-delivery by measuring usage of the screen and audio output devices and test
-it through a survey completed by 47~experiment participants. Unfortunately,
-our results are inconclusive and open to several possible interpretations
-which we discuss.
-
-\newpage
-
-\subsection{Total Energy}
-
-\input{./figures/tables/tableALL.tex}
-
-Clearly, ranking apps by total energy consumption computed by adding all 
-foreground and background energy consumptions over the entire study says 
-much more about app popularity than it does about anything else.
-Table~\ref{table-total} shows the top and bottom energy-consuming apps over
-the entire study. As expected, popular apps such as the Android Browser,
-Facebook, and the Android Phone component consume the most energy, while the
-list of low consumers is dominated by apps with few installs. This table does
-serve, however, to identify the popular apps in use by \PhoneLab{}
-participants, and as a point of comparison for the remainder of our results.
-
-\subsection{Power}
-
-Computing each app's power consumption by scaling their total energy usage
-against the total time they were running, either in the background or
-foreground, reveals more information, as shown in Table~\ref{table-rate}. Our
-results identify Facebook Messenger, Google+, and the Super-Bright LED
-Flashlight as apps that rapidly-consume energy, while the Bank of America and
-Weather Channel apps consume energy slowly. Differences between apps in
-similar categories may begin to identify apps with problematic energy
-consumption, such as contrasting the high energy usage of Facebook Messenger
-with other messaging clients such as WhatsApp, Twitter, and Android
-Messaging.
-
-\subsection{Foreground Energy Efficiency}
-
-Isolating the foreground component of execution time provides a better
-measure of value, since it ignores the time that users spend ignoring apps.
-Table~\ref{table-foreground} shows a measure of energy efficiency computed by
-%utilizing foreground time alone as our value measure. 
-dividing total foreground energy consumption by total foreground time of an 
-app. Some surprising changes
-from the power results can be seen. A number of apps have remained in their former
-categories: Bank of America, which was identified as a low-power app, is also
-a highly-efficient app when using foreground time as the value measure; and
-Facebook Messenger, which was identified as a high-power app, is also marked
-as inefficient. Other apps, however, have switched categories. ESPN
-Sportscenter and Yahoo Mail do not consume much power, but also don't spend
-much time in the foreground; interestingly, none of the high-power apps
-looked better when their foreground usage was considered.
-
-\subsection{Content Energy Efficiency}
-
-Finally, we use the data we collected by instrumenting the
-\texttt{SurfaceFlinger} and \texttt{AudioFlinger} components to compute a
-simple measure of content delivery. We measure the audio and video frame
-rates and combine them into a single measure by using bit-rates corresponding
-to a 30~fps YouTube-encoded video and 128~kbps two-channel audio, with the
-weights representing the fact that a single frame of video contains much more
-content than a single sample of audio. We use this combined metric as the
-value measure and again use it to weight the energy consumption of each app,
-with the results shown in Table~\ref{table-content}.
-
-Comparing with the foreground energy efficiency again shows several
-interesting changes. Yahoo Mail, which foreground energy efficiency marked as
-inefficient, looks more efficient when content delivery is considered. While
-it is possible that one \PhoneLab{} participant uses it to read email very
-quickly, it may be more likely that it uses a ``spinner'' or other fancy UI
-elements that generate artificially high frame rates without delivering much
-information. The inability to distinguish between meaningless and meaningful
-video frame content is a significant weakness of this simple approach.
-YouTube and Candy Crush Saga both earn high marks, which is encouraging given
-that they are very different apps but also might be a result of overweighting
-screen refreshes. The Android Clock is also an unsurprising result, as it
-requires almost no energy to generate a relatively-large number of screen
-redraws in timer and stopwatch mode.
-
-\subsection{Survey Results and Discussion}
-
-\begin{figure*}[t]
-\centering
-\includegraphics[width=\textwidth]{./figures/survey.pdf}
-
-\caption{\small \textbf{Survey Results.} The height of each bar demonstrates how
-many of the suggested apps the user is willing to remove for better battery
-life, with suggestions based on overall usage or our new content-delivery
-efficiency measure. Our new measure does not convincingly out-perform the
-straw man.}
-
-\label{fig-survey}
-
-\end{figure*}
-
-To continue the evaluation of our simple content-based value measure, we
-prepared a survey for the 107~\PhoneLab{} participants who contributed data
-to our experiment. Our goal was to determine if users would be more willing
-to remove inefficient apps, as defined using our content-based measure. As a
-baseline, we also asked users about the apps that consumed the most energy.
-We used each participants data to generate a custom survey containing
-questions about 9 apps: the 3 least efficient apps as computed by our
-content-based value measure, the 3 apps that used the most energy on their
-smartphone during the experiment, and 3 apps chosen at random. For each we
-asked them a simple question: ``If it would improve your battery life, would
-you uninstall or stop using this app?'' To compute an aggregate score for
-both the content-based and usage based measures, we give each measure 1~point
-for a ``Yes'', 0.5~points for a ``Maybe'' and 0~points for a ``No''.
-47~participants completed the survey, and the results are shown in
-Figure~\ref{fig-survey}. For each user, if the score of one measure is higher 
-than the other, it is considered a ``win'' for the former.
-
-Overall the results are inconclusive, with the content-delivery measure not
-clearly outperforming the straw-man usage measure at predicting which apps
-each user would be willing to remove to save battery life. Given the crude
-nature of our metric, this is not particularly surprising, and can be
-interpreted as a sign that we need a more sophisticated value measure
-incorporating more of the potential inputs we have previously discussed.
-However, on one level the results are very encouraging: most users were
-willing to consider removing one or more apps if that app would improve their
-battery lifetime. Clearly, users are making this decision based on some idea
-of each app's value---the challenge is to replicate their choices using the
-information we have available to us.