Curriculum Vitae

Publications

Research Articles

Mejova, Y., Gracyk, T., & Robertson, R. E. (2022). Googling for abortion: Search engine mediation of abortion accessibility in the United States. Journal of Quantitative Description: Digital Media, 2. DOI: 10.51685/jqd.2022.007.

Metaxa, D., Park, J. S., Robertson, R. E., Karahalios, K., Wilson, C., Hancock, J., & Sandvig C. (2021). Auditing algorithms: Understanding algorithmic systems from the outside in. Foundations and Trends in Human–Computer Interaction 14(4), 272-344. DOI: 10.1561/1100000083.

Robertson, R. E., Olteanu, A., Diaz, F., Shokoui, M., & Bailey, P. (2021). "I can't reply with that": Characterizing problematic email reply suggestions. In Proceedings of the 2021 ACM Conference on Human Factors in Computing Systems (CHI 2021). DOI: 10.1145/3411764.3445557.

Ognyanova, K., Lazer, D., Robertson, R. E., & Wilson, C. (2020). Misinformation in action: Fake news exposure is linked to lower trust in media,higher trust in government when your side is in power. Harvard Kennedy School Misinformation Review. DOI: 10.37016/mr-2020-024.

Robertson, R. E., Tran, F. W., Lewark, L. N., & Epstein, R. (2018). Estimates of non-heterosexual prevalence: The roles of anonymity and privacy in survey methodology. Archives of Sexual Behavior. 47(4), 1069-1084. DOI: 10.1007/s10508-017-1044-z

Epstein, R., Robertson, R.E., Lazer, D., & Wilson, C. (2018). Suppressing the search engine manipulation effect (SEME). In Proceedings of the ACM: Human-Computer Interaction, 1(2), Article 42. DOI: 10.1145/3134677

Epstein, R., Ho, M., Hyun, S., Le, C., Robertson, R. E., & Stout, D. (2017). A DSM-5-based online mental health referral inventory: A large-scale validation study. Journal of Technology in Human Services, 35(3), 231-246. DOI: 10.1080/15228835.2017.1356800

Epstein, R., Mejia, J., & Robertson, R. E. (2017). The frequency profile: An informative method for graphing the behavior of individuals post-hoc or in real time. Behavior Analysis: Research and Practice, 17 (1), 55-73. DOI: 10.1037/bar0000052

Epstein, R., Robertson, R. E., Smith, R., Vasconcellos, T., & Lao, M. (2016). Which relationship skills count most? A large-scale follow-up study. Journal of Couple and Relationship Therapy, 15 (4), 341-356. DOI: 10.1080/15332691.2016.1141136

Epstein, R., & Robertson, R. E. (2015). The search engine manipulation effect and its possible impact on the outcomes of elections. Proceedings of the National Academy of Sciences, 112 (33), E4512-E4521. DOI: 10.1073/pnas.1419828112

Epstein, R., & Robertson, R. E. (2014). How to measure sexual orientation range, and why it's worth measuring. Journal of Bisexuality, 14, 392-404. DOI: 10.1080/15299716.2014.933378

Conference Proceedings

Hanyu Chwe, Apryl Williams, Ronald Robertson (2023) "Hey Beautiful": Race and gender on Tinder. In Proceedings of the 56th Hawaii International Conference on System Sciences [PDF].

Jiang, S., Robertson, R. E., & Wilson, C. (2020). Reasoning about political bias in content moderation. In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI 20). New York, New York, February, 2020. (This is an abbreviated version of the "Bias Misperceived" paper that appeared at ICWSM 2019.) [PDF].

Robertson R. E., Jiang, S., Lazer, D., & Wilson, C. (2019). Auditing autocomplete: Recursive algorithm interrogation and suggestion networks. In Proceedings of the 11th ACM Conference on Web Science (WebSci 2019) (pp. 235-244). DOI: 10.1145/3292522.3326047.

Jiang, S., Robertson, R. E., & Wilson, C. (2019). Bias misperceived: The role of partisanship and misinformation in YouTube comment moderation. In Proceedings of the International AAAI Conference on Weblogs and Social Media (ICWSM 2019), 13(01), 278-289. Outstanding Analysis Award. [PDF].

Sapiezynski, P., Zeng, W., Robertson, R. E., Mislove, A., & Wilson, C. (2019). Quantifying the impact of user attention on fair group representation in ranked lists. In Companion Proceedings of the 2019 Web Conference (WWW 2019 Companion) (pp. 553-562). DOI: 10.1145/3308560.3317595.

Hu, D., Jiang, S., Robertson, R. E., & Wilson, C. (2019). Auditing the partisanship of Google Search snippets. In Proceedings of the 2019 Web Conference (WWW 2019) (pp. 693-704). DOI: 10.1145/3308558.3313654.

Robertson, R. E., Jiang, S., Joseph, K., Friedland, L., Lazer, D., & Wilson, C. (2018). Auditing partisan audience bias within Google Search. In Proceedings of the ACM: Human-Computer Interaction, 1(2), Article 1. DOI: 10.1145/3274417. [Best Paper: Honorable Mention] [PDF]

Robertson, R. E., Lazer, D., & Wilson, C. (2018). Auditing the personalization and composition of politically-related search engine results pages. In Proceedings of the 2018 World Wide Web Conference on World Wide Web (WWW 2018) (pp. 955-965). DOI: 10.1145/3178876.3186143

Commentary

Ronald E. Robertson (2022) Uncommon yet consequential online harms. Journal of Online Trust and Safety, 1 (3). 10.54501/jots.v1i3.87.

Pasquetto, I., Swire-Thompson, B., Amazeen, M.A., Benevenuto, F., Brashier, N.M., Bond, R.M., Bozarth, L.C., Budak, C., Ecker, U.K.H. , Fazio,L.K., Ferrara, E., Flanagin, A.J., Flammini, A., Freelon, D., Grinberg, N., Hertwig, R., Jamieson, K.H., Joseph, K., Jones, J.J., Garrett, K., Kreiss,D., McGregor, S., McNealy, J., Margolin, D., Marwick, A., Menczer, F., Metzger, M. J., Nah, S., Lewandowsky, S., Lorenz-Spreen, P., Ortellado,P., Pennycook, G., Porter, E., Rand, D. G., Robertson, R. E., Tripodi, F., Vosoughi, S., Vargo, C., Varol, O., Weeks, B. E., Wihbey, J., Wood, T. J.,& Yang, K. C. (2020). Tackling misinformation: What researchers could do with social media data. Harvard Kennedy School Misinformation Review. DOI: 10.37016/mr-2020-49.

Reports

Cryst, E., Schechter, N., Robertson, R. E., & Thiel, D. (2021). Maduro's menagerie: Twitter client usage in a Venezuelan Twitter operation. Stanford Internet Observatory. Report PDF.

Chen, A. Y., Nyhan, B., Reifler, J., Robertson, R. E., & Wilson, C. (2021). Exposure to Alternative & Extremist Content on YouTube. Anti-Defamation League. Report PDF.

Lewandowsky, S, Smillie, L., Garcia, D., Hertwig, R., Weatherall, J., Egidy, S., Robertson R. E., O’Connor, C., Kozyreva, A., Lorenz-Spreen, P.,Blaschke, Y., & Leiser, M. (2020) Technology and democracy: Understanding the influence of online technologies on political behaviour and decision-making. Joint Research Centre (European Commission). DOI: 10.2760/709177.

Preprints

Robertson, R. E., Green, J., Ruck, D., Ognyanova, K., Wilson, C., & Lazer, D. (2021). Engagement outweighs exposure to partisan and unreliable news within Google Search. Arxiv. ArXiv:2201.00074.

Robertson, R. E., Green, J., Ruck, D., Ognyanova, K., Wilson, C., & Lazer, D. (2021). Engagement outweighs exposure to partisan and unreliable news within Google Search. Arxiv preprint: ArXiv:2201.00074.

Chen, A. Y., Nyhan, B., Reifler, J., Robertson, R. E., & Wilson, C. (2023). Subscriptions and external links help drive resentful users to alternative and extremist YouTube channels. Arxiv preprint: ArXiv:2204.10921.

Work in Progress

Gleason, J., Hu, D., Robertson, R.E., & Wilson, C. (under review). Google the gatekeeper: How search components affect clicks and attention.

Robertson, R.E., Dunphy, A. & Grossman, S. (under review). Identifying search directives on social media.

Lewandowsky, S., Robertson, R.E., & DiResta, R. (under review). Challenges in understanding human-algorithm entanglement.

Hu, D., Gleason, J., Abu Bakar Aziz, M., Guggenberger, N., Robertson, R.E., & Wilson, C. (under review). Google Search dominates horizontal search and most vertical search segments on the desktop.

Hu, D., Robertson, R.E., Hannak, A., & Wilson, C. (under review). Exposure to YouTube videos on- and off-platform.

Media Articles

Robertson, R. E., Jiang, S., Joseph, K., Friedland, L., Lazer, D., & Wilson, C. (2018, November 6). Is it the Algorithms or Us?. Medium

Robertson, R. E. (2018, June 4). When will Google defend democracy?. The Conversation

Robertson, R. E. (2018, January 15). The 21st century Skinner box. Behavioral Scientist

Software

Robertson, R. E. (2019). WebSearcher: Tools for auditing web search. pip package, version 0.1.14.

Robertson, R. E. (2019). suggests: Tools for auditing autocomplete. Python package, version 0.1.0.

Robertson, R. E. (2016). dtables: Simplifying Descriptive Frequencies and Statistics. CRAN package, version 0.2.0.

Robertson, R. E., Tran, F. W., Mejia, J., & Mourani, C. (2016). IPtoCountry: Convert IP Addresses to Country Names or Full Location with Geoplotting. CRAN package, version 0.0.1. 2016 IP2Location Programming Contest Winner

Epstein, R., Boulier, T., & Robertson, R. E. (2016). FreqProf: Frequency Profiles Computing and Plotting. CRAN package, version 0.0.1.

Robertson, R. E. (2016). files: Effective File Navigation from the R Console. CRAN package, version 0.0.1.

Presentations

Robertson, R.E., Wilson, C., & Lazer, D. (2019, August). Online Information Seeking during the 2018 U.S. Congressional Elections. Paper presented at the 2nd annual Politics and Computational Social Science (PaCSS) conference. Washington, D.C.

Robertson, R.E., Wilson, C., & Lazer, D. (2019, July). Fake News and Information Seeking during the 2018 U.S. Congressional Elections. Paper presented at the 5th International Conference on Computational Social Science (IC2S2). Amesterdam, Netherlands.

Robertson, R.E., Jiang, S., Lazer, D., & Wilson, C. (2019, July). Auditing autocomplete: Recursive algorithm interrogation and suggestion networks. Paper presented at the 11th ACM Conference on Web Science (WebSci 2019). Boston, MA.

Robertson, R.E., Jiang, S., Jospeh, K., Friedland, L., Lazer, D., & Wilson, C. (2018, November). Auditing partisan bias within Google Search. Paper presented at the annual meeting of the association for Computer-Supported Cooperative Work (CSCW 2018). Jersey City, NJ.

Robertson, R.E., Lazer, D., & Wilson, C. (2018, April). Auditing the Personalization and Composition of Politically-Related Search Engine Results Pages. Paper presented at the 2018 World Wide Web Conference (WWW 2018). Lyon, France.

Robertson, R.E., Lazer, D., & Wilson, C. (2018, March). Modeling suggestion networks for algorithm auditing. Paper presented at CompleNet 2018. Boston, MA.

Epstein, R., Mourani, C., Olson, E., & Robertson, R. E. (2017, April). Biased search rankings can shift opinions on a wide range of topics. Paper presented at the 97th annual meeting of the Western Psychological Association. Sacramento, CA.

Epstein, R., Ding., M., Mourani, C., Olson, E., Robertson, R. E., & Tran, F. (2017, April). Multiple searches increase the impact of the Search Engine Manipulation Effect (SEME). Paper presented at the 97th annual meeting of the Western Psychological Association. Sacramento, CA.

Epstein, R., & Robertson, R.E. (2017, March) The Search Engine Manipulation Effect (SEME): Understanding its power to change opinions and votes. Paper presented at the 2nd biennial meeting of the International Convention of Psychological Science. Vienna, Austria.

Epstein, R., & Robertson, R. E. (2016, April). A replication of the Search Engine Manipulation Effect (SEME), plus methods for suppressing the effect. Paper presented at the 96th annual meeting of the Western Psychological Association. Long Beach, CA.

Epstein, R., & Robertson, R. E. (2016, April). Why is the search engine manipulation effect (SEME) so large? Testing an operant conditioning hypothesis. Paper presented at the 96th annual meeting of the Western Psychological Association. Long Beach, CA.

Epstein, R., Vu, K., & Robertson, R. E. (2016, April). Which skills are best at boosting motivation? A large-scale Internet-based study. Paper presented at the 96th annual meeting of the Western Psychological Association. Long Beach, CA.

Epstein, R., Nanayakkara, P., & Robertson, R. E. (2016, April). Which parenting skills count most? A large-scale Internet study. Paper presented at the 96th annual meeting of the Western Psychological Association. Long Beach, CA.

Epstein, R., Robertson, R. E., & Hyun, S. (2016, April). Calling oneself straight can be stressful: Insights from a large multinational study of sexual orientation. Paper presented at the 96th annual meeting of the Western Psychological Association. Long Beach, CA.

Epstein, R., & Robertson, R. E. (2015, April). The Search Engine Manipulation Effect (SEME): Large-scale replications in two countries. Paper presented at the 95th annual meeting of the Western Psychological Association, Las Vegas, NV.

Epstein, R., Le, C., & Robertson, R. E. (2015, April). A DSM-5-based online mental health screening inventory: Preliminary validation study. Paper presented at the 95th annual meeting of the Western Psychological Association, Las Vegas, NV.

Epstein, R., Thompson, D. S., Crawford, A. S., Mejia, J., & Robertson, R. E. (2015, April). The frequency profile: An informative method for graphing the behavior of individuals post hoc or in real time. Paper presented at the 95th annual meeting of the Western Psychological Association, Las Vegas, NV.

Epstein, R., Hwang, T., & Robertson, R. E. (2015, April). Extended Childhood Disorder (ECD): Additional support for a new diagnostic category. Paper presented at the 95th annual meeting of the Western Psychological Association, Las Vegas, NV.

Epstein, R., & Robertson, R. E. (2014, April). Helping people preserve their privacy online: The surprising power of a click requirement. Paper presented at the 94th annual meeting of the Western Psychological Association, Portland, OR.

Robertson, R. E. & Epstein, R. (2014, April). Are we underestimating non-heterosexual prevalence? The critical role of survey methodology. Paper presented at the 94th annual meeting of the Western Psychological Association, Portland, OR.

Epstein, R., & Robertson, R. E. (2014, April). How adultness varies across the life span: A large-scale Internet study. Paper presented at the 94th annual meeting of the Western Psychological Association, Portland, OR.

Epstein, R., & Robertson, R. E. (2013, November). A quantitative analysis of the mismatch between sexual orientation labels and various measures of sexual expression. Paper presented at the 56th annual meeting of the Society for the Scientific Study of Sexuality, San Diego, CA.

Epstein, R., & Robertson, R. E. (2013, November). How to measure sexual orientation range and why it's worth measuring. Paper presented at the 56th annual meeting of the Society for the Scientific Study of Sexuality, San Diego, CA.

Epstein, R., & Robertson, R. E. (2013, November). How to measure sexual orientation prevalence without underestimating the prevalence of non-heterosexuals. Poster presented at the 56th annual meeting of the Society for the Scientific Study of Sexuality, San Diego, CA.

Epstein, R., & Robertson, R. E. (2013, May). Democracy at risk: Search rankings can shift voter preferences substantially. Poster presented at the 25th annual meeting of the Association for Psychological Science, Washington, D.C.

Epstein, R., & Robertson, R. E. (2013, April). The inadequacy of sexual orientation labels: Lessons from a large-scale study. Paper presented at the 93rd annual meeting of the Western Psychological Association, Reno, NV.