Category: Quant Humanists

Digital phenotyping + self-care/intervention

• Background research/ project landscape:
  • Mindstrong: This is basically what I came to ITP to do: develop a tracking system that can detect/predict/prevent the onset of depression, although Mindstrong is aiming to tackle many mental illnesses with only smartphone data, whereas I am primarily gathering data from laptop use. Co-founded by Dr. Tom Insel, formerly the lead of Verily’s mental health team.
  • PRIME app: an app developed by UCSF researchers for clinical trials studying the effect of social support on the severity on schizophrenia
  • Fine: An mood reporter app that tracks self-reported data (not available for use)
  • trackyourhappiness.org: an ongoing doctoral research project by Matt Killingsworth at Harvard, which prompts you throughout the day to find out what factors are associated with happiness.
  • PHQ-9: standard self-reported questionnaire for depression severity
  • Exist: links all your tracking apps to find correlations
  • I feel like shit game: an interactive self-care flow chart; asks you questions about your state and offers self-care suggestions
  • Headspace: there’s a meditative exercise tailored for nearly every mood possible

• Hypothesis / Definition of question(s):
  • What factors in my life contribute to stress, anxiety, low morale/motivation, and negative affect in general?
  • What factors contribute to high morale, motivation, positive affect, and a more balanced feeling of well-being?
  • How might I facilitate the latter factors?
  • What interventions are appropriate?
• Objectives:
  1. a system that tracks:
  2. a dashboard for data viz
  3. machine-learned correlations (ultimately)
  4. self-care recommendations
• Goals:
  • To address issues with current treatments:
    • Therapy:
      • lag time (“A therapist, the joke goes, knows in great detail how a patient is doing every Thursday at 3 o’clock.”)
      • No performance feedback outside of potentially dishonest self-reporting
      • patient fear of disappointing therapist
      • variations in individual therapist efficacy
    • Medication:
      • crude; not even sure how it works
      • mental health is a social issue
  • To catalyze self-awareness of emotions and their triggers
  • To facilitate self-care
  • To encourage healthier digital habits
• Technical considerations/next steps:
  • Finish trackers:
    • tab/window counter: chrome extension
    • new affectiva approach: chrome extension background page?
    • unreturned messages
    • self-reported mood
  • Research visualizations
  • Research more behavioral metrics

SLIDES HERE

So my keylogger has been talking hourly to IBM’s tone analyzer API for a couple weeks now, and I’ve been noticing strange responses to the chunks of time I’ve spent coding:

When I first noticed this behavior, I thought it amusing yet unforgivably flawed; I was trying to do some Serious Sentiment Analyses here, and clearly I was not experiencing such extreme mood swings while programming (albeit, the examples above illustrate my experience pretty well).

But in the context of this assignment, it provides an easy opportunity for manipulation— I just needed to figure out what exactly was triggering these false positives. So I gathered all the erratic predictions and fed them “word” by word into IBM’s demo, adding and deleting until I found precisely which characters the model was responding to. For the foregoing examples:

A few other absurd samples:

I also found that several individual words triggered extremely confident predictions:

So what can be used to consistently hack IBM’s predictions?

For starters, any built-in function (that’s also a complete word) will get picked up for displaying confidence:

Even when followed by text that’s obviously not-so-confident:

Even when in nonsensical function salad:

To sound analytical, one must simply add the word “if” anywhere in a sentence:

Regardless of whether or not the surrounding words are analytical:

To seem instantly and dramatically happier, just add {}:

Curly brackets are so joyful that they even neutralize fear and sadness:

Lastly, the singular most effective way to express anger is with this emoji: =P

Its rage is so complete that it literally sucks the joy out of life:

I’ve been in grave danger ever since discovering Sex and the City on Amazon Video.

I normally avoid all media like the plague, partially because I find the content objectionable for SJW reasons, partially because of my own insecurities which I will probably intellectualize away with SJW reasons forever, but mostly because I can’t trust my addictive personality to watch even one video without immediately downward-spiraling for ten hours into a pit of shame and self-loathing.

The one exception I make is for SATC, for no good reason other than the fact that I was able to cobble together the entire series on the cheap during my phase as a Housing Works regular (this is generally how I consume media: once everyone’s completely over it). Despite this DVD collection’s efficacy as a coping mechanism, its inconvenient—and now obsolete, thanks to Apple’s sanctimony— physical form was never a threat to my daily functioning.

Until:

By some loving grace of God, I only discovered this year that SATC was included with Amazon Prime, but somehow I’ve already watched three seasons of it, plus the first movie, and 1.5 seasons of SJP’s new show, Divorce, which I highly do not recommend, and only watched while nursing the sugar headache that SJP’s younger self tends to cause (I’m Team Kim).

So this week, I decided to put an end to this nonsense. RescumeTime, a tracker that I installed near the beginning of QH, converts my activity into a handy “productivity” score—one that I get to define by categorizing any website or application I use on a scale from “Very Productive” to “Very Distracting”:

I decided to use python to grab this score every hour through RescueTime’s API, and open my slightly meditative, mostly masochistic, brain entraining p5 sketch if the score dips under 50%:

Here it is in action:

The idea of this intervention is to allow myself the room to indulge in “very distracting” activity if I need it, but to catch myself before I spiral out of control and have to live with the concomitant guilt forever. The “entrainment” part—regardless if it actually entrains my brain or not—is, at the very least, a way of resetting myself and my OCD.

This d3 sketch illustrates how my activity changes before and after “entrainment”, and I’m working on automating the chrome history collection so that I can have a viz that automatically updates in real(-ish) time.

JENNA XU
QUANT HUMANISTS
SPRING 2018
26/02/2018

Several sources of inspiration for this week’s assignment:

Humanist service design guidelines: Ie, rather than keeping users hooked on a product, we should be designing products that facilitate our human needs: sleep, human connection, engagement with the physical world, etc. By the Center for Humane Technology.

This Vox article also galvanized me to reconsider the importance of social connection on mental illness, while reminding me of the town of Geel, which treats mental disorders with social inclusion.

Anytime you consider anything less than everything you are missing something: J. Paul Neeley’s beautiful talk on considering everything really validated my insatiable impulse to collect the data possible.

JENNA XU
QUANT HUMANISTS
SPRING 2018
26/02/2018

Persona:

User Experience Journey:

Service Blueprint:

Prototype touchpoint

For week 2’s assignment, my boyfriend and I logged our feelings every hour for about four days. We had recently realized just how differently we perceived and experienced the same relationship, and thought it would be interesting to do a comparison.

The variables we manually tracked on a Google spreadsheet were time, reaction, whether it was a positive or negative feeling, description of the event that triggered the reaction, and an overall “satisfaction score”:

On Monday, he forfeited his log and I mapped both datasets on the postcards Matthew gave us in class:

First dataset is his, bottom is mine

I choose a simple bar graph in order to flatten an extremely nuanced and qualitative dataset into something visually digestible. The bars were encoded with two colors: pink for the score, and a secondary color that indicated the type of trigger. The fluctuations in bar height in my boyfriend’s postcard illuminated just how much my boyfriend’s anxiety and sensitivity affects his experience of our relationship, while my rather uniform results illustrated how generally unperturbed I am, and/or how oblivious I am to his emotions.

JENNA XU
QUANT HUMANISTS
SPRING 2018
02/05/2018

This week, I learned that I am totally ADD and greedy and want all the data. I also learned that all the data will definitely be too much data, and decided I will not be getting an iPhone just to use a select few iOS-only apps, so I’m going to use this space to determine my final line-up of trackers. Here goes:

1. The “how are you feeling?” trackers:
   • Affectiva, as we saw below, which uses computer vision and a webcam to translate facial markers to emotions. Will be using this to track my blinking as well.
   • IBM’s Tone Analyzer and a key logger for a sentiment analysis keyboard. Will probably transfer text and format data manually for now, until I get time to set up something more automated.
2. The “is your mind wandering/lonely?” trackers:
   • Affectiva calculates attention, engagement, and valence values, in addition to blinks
   • HabitLab, by Stanford HCI Group, to track my insidious social media usage, as well as to build better habits (might switch to RescueTime if this doesn’t suit my needs, but HabitLab had a much more attractive UI)
3. The “what makes you happy?” tracker:
   • The Track Your Happiness app is only available of iOS, but it’s simple enough that I may be able to build something similar if I have time (or use this: https://www.askmeevery.com/)
   • Activity tracker: Google maps/pedometer in smartphone
   • Weather
   • Social activities

JENNA XU
QUANT HUMANISTS
SPRING 2018
05/02/2018

Forget all the trackers I wrote about last week. I’ve met someone new.

MIT’s Affectiva uses computer vision/machine learning to interpret facial biometrics into emotions, in real time, through webcam input. I downloaded a browser demo to play around with and see what the data would look like. Happily, the SDK spits it all out in a nice JSON format:

The big question was figuring out what data to collect, and how frequently. The demo seemed to be returning data for every frame—way too granular, especially considering that I intend to collect on the long-term.

After some experimentation, I decided to only record the emotion variables that reached a value of 95 out of 100, and the expression variables that reached 99.5 out of 100 (these were more sensitive). With each of these, I also pushed the values for attention, valence, engagement—because I’m most interested in tracking mind-wandering—as well as the “dominant emoji” and a timestamp. I figured this would give me a pretty good picture of my mood shifts throughout the day, at a reasonable pace.

Well, after a mere hour or two, my laptop fans were going at full speed, and a preliminary download of the data looked like this:

Existing quietly at a rate of 493,283 JSON columns / hour. 

To test the physical limits of my laptop, I decided to throw this thing into a d3 sketch:

NBD, just a webpage with 16,000 DOM elements. This is going well.

Also, just kidding about the other trackers, I still plan on using/hacking many of them. I just got a little uh, sidetracked this week. I also tried out the Beyond Reality js face tracking library, which was very impressive, but Affectiva can do everything it does and more. 😍

JENNA XU
QUANT HUMANISTS
SPRING 2018
29 JAN 2018

Write a short reflection about what your current relationship with self-tracking (e.g. hopes, dreams, perceptions), questions you have about self-tracking and how it could help or harm you, and how you hope the course will help facilitate your interests. Write about which questions you’ve identified to track, how you plan to track those variables of interest, and what challenges you expect to encounter as well as what you hope to learn.

Out of lack of self-discipline, I don’t currently have a relationship with self-tracking, but at ITP I ultimately hope to track the biometrics of depression and its related factors, and feed this data into a neural network that can predict my moods and help monitor the cycles of depression. I hope this course will allow me to explore my options for tracking these variables and perhaps even integrate them into an automated system of my own making. Below is my wish-list of questions/methods that I want to work on for this course:

1. What is your current mood? (based on metrics)
   • • https://arxiv.org/pdf/1711.06134.pdf
       This study uses a smartphone app to calculate happiness based on psychology’s “traditional view of happiness as a parameter of two dimensions: arousal and valence”. Arousal is measured based on data coming in from a smartwatch, which tracks activity, heart rate, light level, and GPS coordinates. They also consider external factors such as weather and day of the week. By asking for self-reported mood data, they were able to input this data into a machine learning-based system (random forests) to predict mood.
     • https://tech.cornell.edu/news/this-smartphone-keyboard-app-can-read-your-emotions
       This project tracks variables like typing speed, punctuation changes, and sentiment analysis of what’s being typed to detect the user’s emotions.
       • https://github.com/Keymochi
2. What makes you happy? (based on self-reporting)
   • https://www.trackyourhappiness.org/
     This is an ongoing doctoral research project by Matt Killingsworth at Harvard, which prompts you throughout the day to find out what factors are associated with happiness.
3. Is your mind wandering?
   • Mind wandering and rumination is a hallmark trait of depression, and is associated with increased blinking. Here’s a javascript library that can detect blinks:
     • https://tastenkunst.github.io/brfv4_docs/what_can_i_do_with_it.html
   • https://www.rescuetime.com/ 
     To track random browsing/productivity
   • Will do more research on this one
4. Do you feel lonely?
   • Depressed people spend more time on social media, which was true for me as well. Though I’m not active on anything presently, I still occasionally catch myself navigating to Facebook or Instagram when I feel agitated; this probably is indicative of loneliness—something to investigate!
     • Here’s a chrome extension to track FB usage: https://chrome.google.com/webstore/detail/time-spend-on-facebook/cbdiihnkhjaiokcaeiecemajlohbhefo?hl=en
     • An app for app tracking: https://inthemoment.io/
   • http://fetzer.org/sites/default/files/images/stories/pdf/selfmeasures/Self_Measures_for_Loneliness_and_Interpersonal_Problems_UCLA_LONELINESS_REVISED.pdf
     • This is a self-reported questionnaire on loneliness that could be used similar to trackyourhappiness

My main challenge will be to actually implement any of this! Most of these apps are from academic studies and not available to the masses, and the ones that are are only for iOS.