Amy Choi [AC]: All right guys, thank you so much for being here today. In this panel we are talking about information. How we gather it, how we understand it, how we deploy it in our work, and we have 2 amazing people here for the conversation. Paco Viñoly from Nextdoor, and Anne Diaz from Airbnb. Hey guys thank you for being here.

Anne Diaz [AD]: Hi.

Paco Vinoly [PV]: Hello, hi.

AC: So I wanna start off by thinking a little bit about how we even think about questions and information gathering. How do we ask the right questions and how do we create the best possible products and experiences with the data that we gather. Like kind of set the stage for how our own perspective and our own world views like really deeply influence the kind of questions we ask and then again – on the flip side, how we interpret it. So 2 examples come to mind.

One is, oh the great embarrassment of the Pepsi ad that just came out. Pepsi created what is universally being lauded – hm. Sarcastic term – as a completely tone deaf advertisement. So essentially it was how Pepsi’s gonna solve the social justice movements of the world by having Kendall Jenner give some riot police a can of Pepsi. And so this to me is like the most recent example of how you can take the data that you have – the information at hand and create what you believe is the best possible product with that data – and still be completely off because of missteps all along the way. So to a brand, they could be like what is the hottest thing right now? Kendall Jenner is the hottest celebrity. Social justice it the topic of conversation and the – the rising political activism of young people. And then millennials. And then they also knew that they needed to be diverse, right? Like everybody knows that diversity is a thing that we should care about. And so they created an ad in which there’s Kendall Jenner, there’s young people. They’re protesting something – you know there’s a lot of nods to diversity. There’s a young woman in hijab, there’s black kids dancing, there’s Latinos, there’s white people. And yet and yet with all of that data they created a big mistake which they have apologized for in full. So here for me is an example of how you can believe you have all the right information at hand, and still you’re not interpreting it the right way because of your perspective, because of people who are not sitting in the room with you.

Another example that I was recently discussing with Aminatou Sow, who we’ll hear from a little bit later in the show – she does a lot of work with Sundance and she was recently in conversation with a producer who was talking about how you know – studios – film studios make decisions based on data. You know they make decisions and execute based on all of this information and the algorithms that they apply to it. And one of the data points is that women don’t create blockbuster action films. These big budget Star Wars like film. And the data has borne out that there have been no successful examples. But the point that Aminatou is making is that the reason that there’s no data that bears that out is because no woman has been given an opportunity to direct big budget film. So you know there’s all these different steps at which your perspective can really shape the decisions that you’re making. So that’s just kind of to set the frame for this conversation. You know like information data is the best. We all like kind of bow to the god of information unless we’re gathering it the wrong way and then looking at it wrong. That can really lead to some missteps.

So – Anne, you’re a researcher. How do we avoid these kinds of pitfalls? Like how do we think inclusively from the get-go as we’re gathering information?

AD: Yeah so I was thinking about this a little bit last night and just trying to think through – you know when I start on a research project how I try to approach it to make sure that I’m thinking from all the different angles that are gonna be helpful to answering this question. And I think the strategy that I try to use is figuring out what I want to do with the data that I find. So not just looking at data as like this treasure trove of information that is somehow going to tell me exactly what to do. But instead thinking about like what’s the North Star, what’s the thing that I want to do with what I learn from this information. And being really really explicit about that and upfront about it. And getting the other people that I work with to also buy into yes, this is what we wanna do with this data once we gather it. By having that kind of North Star, it really helps to number 1, figure out who you need to talk to, and it also helps to weed out the bias that inevitably creeps in when you start looking at data. You’ll start to find things that are really interesting and that might take you on a tangent. And sometimes those tangents are really beautiful and really meaningful. But I think the most important thing is constantly – especially as a researcher as I’m interviewing people, constantly trying to understand – is what I’m hearing right now, is that gonna help with that ultimate goal that I set up to accomplish.

AC: Right.

AD: And steering the conversation in ways that will help me get information. That ultimately will lead to that goal.

AC: That’s interesting too – that’s a great point – is that having that North Star is so essential. Because you’re not in the business of just getting any information you want. Right or –

AD: Right.

AC: or just you know – like how do you make also the information gathering productive and useful to you.

AD: Mmhm.

AC: How do we question our own assumptions about what is a tangent and then what is actually the real story.

AD: Mm.

AC: You know like –

AD: Yeah.

AC: Does the North Star sometimes shift?

AD: Yeah and I think that’s why it’s really important to come up with some sort of North Star that’s actually not a hypothesis. So –

AC: Mm.

AD: there are probably many researchers who would disagree with me on this point, but instead of saying you know – I wanna prove whether or not this thing happens.

AC: Mmhm.

AD: When I start on a research project I am thinking about what do I need to learn in order to do X? So like we want to build a stronger profile for example on Airbnb. So what do I need to learn that will help me figure out what a strong profile looks like? And –

AC: Right.

AD: really trying to define like where this data is going to go first instead of –

AC: Mmhm.

AD: I think – you know here are all my hypotheses about what people will want. You know sometimes people answer a question and you realize they’re trying to tell you something else. They’re trying to –

AC: Mmhm.

AD: answer a question that maybe wasn’t in the room. You know trying to be very sensitive to that as well, because I think that the best sort of research interviews stem from people being very candid and very honest and you have to be paying attention to what people want to tell you in order to –

AC: Mmhm.

AD: get that real candor to come out.

AC: It’s so much about your EQ – right like

AD: Mmhm.

AC: your emotional intelligence and then also creating a space in which they can feel secure

AD: Exactly.

AC: in actually telling you the truth.

AD: Right.

AC: Paco, I’d like to hear from you about – from the design perspective how design can quickly respond to new information that comes in. I’d love to know kind of the insider perspective on how Nextdoor responded to you know – how the app was being used. Specifically in response to when you when you went through the exercise of like reducing racist posts on the app.

PV: Interesting. The first part of your question I think is interesting because the relationship of design with data is always a complex one. We try to also have a North Star that we’re designing to. But it can be perhaps at times overly humbling and distracting – what the data will say

AC: Mm.

PV: to you. It’s extremely valuable for a designer to react and respond to the data but I think it has to be done in the right context and particularly in the right time frame. Because being reactive isn’t always the right thing. So I think that’s kind of in the –

AC: Right.

PV: something I think a lot about because we do gather a lot of data and a lot of metrics at Nextdoor and making sure that we are still designing for our members is – we have to find a right balance. In terms of the racial profiling it’s a very interesting project we went through because we had designed a system that we thought was good. And it was working. And –

AC: I mean don’t you always?

PV: Yeah exactly.

AC: We all do –

PV: that was-

AC: we’re like oh no this – why would you do it if you didn’t think it was –

PV: Right.

AC: good.

PV: So we thought we had covered everything. And as we launched these things into the wild they start evolving and we start learning. And in some ways we learned as much as we could on our own. And –

AC: Mmhm.

PV: then we got a little help. A little help I think in different data points that informed the design and the product work we were doing. So those data points involved working with the community groups. We focused the work primarily around Oakland, here in California. It’s close to us, and we could learn a lot faster and then be able

AC: Mmhm.

PV: to deploy. So we worked with community groups, we worked with um for example Neighbors for Racial Justice. We worked with

AC: Mmhm.

PV: 100 black men, we worked with the City Council and the city government. We worked with the Police Department. So we worked – we tried to gather as much information and perspectives. It was a difficult process for us because it’s – we were learning things that we weren’t expecting to learn. And a perspective

AC: Mmhm.

PV: that was completely new to us.

AC: Just very challenging topics, too. It’s not like you were learning that people use things on their iPhones instead of on their laptops.

PV: Yeah and we were learning a lot about unconscious bias. Obviously it’s unconscious so we weren’t aware that we had that.

AC: [laughs] Right, right.

PV: And trying to really get help in working through that was super valuable. We iterated –

AC: Mmhm.

PV: and ultimately we came up with a – with a product that has tremendous amount of friction which is not usually what we design for, especially in the –

AC: Right.

PV: the online world. But that friction created benefits and created a positive outcome for our members ultimately, right. So we took all the feedback from the community but ultimately we designed products for our members and the neighbors. And the outcome was something that was positive for them as a tool. It was also positive in that it helped the government, Police Department to be able to react in the right appropriate way.

AC: You have some stats there too. What was the reduction rate?

PV: Yeah you know I don’t have the latest stats- when we launched we lowered the incidence of racial profiling by 75%.

AC: That’s amazing.

PV: Yeah.

AC: It’s unbelievable.

PV: It’s – we’re not gonna solve racism. And at first the initial reaction from a lot of the team was – this is human nature. But as we kind of dug deeper and rolled up our sleeves we realized that if we can make a little bit of a dent in

AC: Mmhm.

PV: in propagating of this behavior. At the scale that we work at, it could be a meaningful difference. And it’s – it became a really important thing for us on the team, for us as a company, because that’s the mission

AC: Mmhm.

PV: that we’re on, is to make our neighborhoods better and stronger. And this was a very tangible way of seeing it come to life.

AC: Mmhm. For our listeners I am nodding vigorously into my microphone. As we say at Mash-Up, we all enter the fight at different levels and with different tools. And we can all have an impact in our own way. So as you said, Nextdoor may not solve racism. But, you have built now something that can have a real use for the communities that are using it. And can truly help them do better in that regard. And I think you raised two really important points here. And the one that I’d like to touch on first is about humility. Of course you only – you can only beta test so much, you need to see your product out in the wild. But we also live in a world in which you know – it feels like all information is readily available. And we all need to be experts in everything. At least very much experts in our field, right? I think what we’re talking about here with humility, with asking questions, with questioning our North Star in a way is about letting ourselves being open and vulnerable – and having some humility. And realizing that like okay, I don’t have to be an expert. Like I can ask more questions and that actually will deepen and make our designs better. Right like that will make what we’re offering better by being a little bit vulnerable. And I just wanna have that in our minds as we kind of go through this conversation. Because it’s super hard to do, right. Like who wants to say that they don’t know something.

PV: Humility became a part of my life as a designer early on. So I was just a little bit pre- the web.

AC: There was a time before the web?!

PV: Yeah.

AC: [laughs]

PV: A long time ago. Our listeners won’t know what I’m saying now but – but there was a time when the designer ego ruled supreme. And when –

AC: Mmhm.

PV: When we started working in digital and in the web and metrics, and – multivariate testing and all these statistics came into play. I think design took a humbling medicine right.

AC: Right.

PV: The next step happened when we started working at such large scale with – for me the experience of social networks is trying to control what millions of people will want to do on your platform is a fruitless task. So working with them and facilitating what naturally happens is another big lesson for me at least in how I approach design.

AC: Yeah that’s such a great point. Anne, how does humility play in designing research?

AD: One of the things I really love about being a researcher is that in order to be a good researcher humility is just the name of the game. I mean I think that the way to be an expert in research is to be really good at saying I don’t know but I want to find out.

AC: Mmhm.

AD: You know constantly challenging your own assumptions and the benefit of being a researcher too, is that you know designers have to take this information and then do something with it. And my role is to just find it. [laughs] To kind of dig it up. And I think one of the things I really enjoy as well about research is that there is something really powerful in looking at the world in the way that it is as opposed to the way that you want it to be. If we can look at something in the eye whether it’s racial profiling or discrimination and we can look at it and just see – this is how it happens. This is what’s going on, these are the motivations that people have and this is what they’re trying to do. You kind of have then all the tools that you need to start to figure out how do we change this, and how do we –

AC: Mmhm.

AD: maybe make a new reality. But you have to start by really being pretty honest about what’s actually going on.

AC: I love that, I love that. And that for me, is right in this moment right now, it feels so profound to say that the first step is looking – looking reality in the eye. And I think that’s certainly been true of our work at The Mash-Up Americans which is that you know we started our exploration of race and culture and identity in America as a celebration

AD: Mmhm.

AC: of all the different things that make us who we are. And in the past couple months it has become a cause. We like to say we – since the election, we know better about the marketplace in which we operate. There’s a large part of that which is looking it in the eye. Looking

AD: Right.

AC: at reality in the eye.

AD: Mmhm.

AC: You know the other deep theme that has arisen from the conversation so far is about community. You know and to me at its heart both Nextdoor and Air and [sic] bnb are about building community in some way. You know Nextdoor is almost very literally you’re talking to your neighbors right? In – you’re talking to your neighbors in a digital way and then seeing them out in the wild in an analog way. And Airbnb is another kind of building community. It’s opening and sharing your home, right. And why do – why are people coming into your home if you’re a host? They’re coming into your city, into your space. They’re seeing how you – how you live in the world really by being inside your home. And I can’t really think of a more – a deeper way in which you build community and like – and make connections. And so it – you know it feels to me like – that both of these are at heart, if we’re here talking about designing solution – about being deeply inclusive. You can’t build community by necessarily being exclusive, although Paco you had a great point that like – some things have to be excluded because if you’re in Oakland then you don’t need to include a neighbor, quote neighbor that’s in LA. Like that not

PV: Right.

AC: part of your community. So we have boundaries around this right? Recently Kristy Tillman at Slack had been quoted in an article saying, “I found myself in places that weren’t designed for me. But surely places where I had a perspective to offer. Surely places that people already at the table needed my perspective unbeknownst to them.” And I think there’s something about that – that unbeknownst to – you don’t know what you don’t know. But then you start actively and proactively building so that you can access more. So that you can be more inclusive and that feels really important in this conversation about gathering data and how you interpret it. Anne, as you – in your experiences, you’re experimenting with different research approaches – what do different strategies reveal? You know Paco was saying a little bit about what happened when the makeup of his team changed. What happens when the makeup of your questions change or – what are you best practices for actively avoiding exclusion as well?

AD: Yeah so one thing that we’ve been doing at Airbnb and being really really thoughtful about this – is of course because we are a travel company, and we are a global company – making sure that our research does not just happen in the Bay Area. The Bay Area is a real –

AC: [laughs]

AD: not the most representative place.

AC: It’s not a good focus group for 1 as we like to say.

AD: Exactly. And you know if the question is about hosts in the Bay Area it’s a fantastic place. But when we talk about hosts in general or let’s say hosts who rent out vacation rentals or even folks who are traveling throughout the United States. We can’t just look at the Bay Area and so while we have some fantastic research labs that are in our office here in San Francisco – we’ve really made a very conscious effort on the research team to get out of the building. And so we’ve been –

AC: Mmhm.

AD: conducting a lot of research that’s remote with folks all over the United States and also outside of the country as well. Many of my colleagues have been taking trips to Europe and to Asia, to Latin America. Places where we have very very high concentrations of people using Airbnb, and we need to learn from them. And so we’ve been partnering with folks in those different countries who can help us with translation and helping to set up focus groups in ways that will be comfortable and helpful. We’ve also been working a lot with our localization team. This year I got to go through a process that- I learned a new word called “trans-creation.”

AC: Trans-creation?

AD: Trans-creation, yeah. So we were working on – I work as part of the anti-discrimination product team at Airbnb. And we launched a community commitment to all of our hosts and all of our guests. So when you go on Airbnb, if any of you have logged in recently you’ve probably seen this. We ask everyone to make a public commitment to working to overcome their own bias and being open and accepting of others in the community. And so the way that we wrote this commitment – we wanted it to be relatively short so that people would read it. But we also wanted it to be meaningful and not to sugarcoat things. So we – but we wrote it with a very American idea in mind. So for example, we – we sort of list out all of these different protective classes that we wanna make sure that you don’t exhibit discrimination towards.

AC: Mmhm.

AD: And the language itself is fairly firm. It sounds a lot like sort of equal employment opportunity language that you might see in other

AC: Right.

AD: places. What we found is that in many other countries, China in particular, this language was not landing very well. So we had just gone through a basic translation process which is almost a word for word translation. We met with the localization team to try and figure out what was going on and they said you know, this is just – it’s really weird in Mandarin. It’s really weird. And so we sat

AC: Mmhm.

AD: down to go through again what’s called a trans-creation process where you take language and you take an idea and you figure out how to make it work for another culture. So instead of –

AC: Right.

AD: just translating word for word, what was really interesting is that in Mandarin the language ended up completely changing to be more about you know – won’t you join us in helping to make the world more harmonious. So it was –

AC: Mm.

AD: much more aspirational and a – and a little bit more

AC: Mmhm.

AD: targeted towards like be a part of this group. And

AC: Mmhm.

AD: the American language was very much about like hey, you should you know do this thing as part of a community, very individual.

AC: Mmhm.

AD: And that was just such an interesting process.

AC: Oh and that’s so American and so Chinese.

AD: Yes exactly exactly.

AC: That makes – of course that makes perfect sense. Oh thank you for sharing that story. I think that there’s so many lessons that are revealed in that. Also in that you know that was the lesson that Airbnb had, too. I’m sure when the service launched it wouldn’t have necessarily occurred to people that like hosts wouldn’t wanna rent to black people. Or to A- like that the is something that has to be responded to.

AD: Mmhm.

AC: As we posited earlier you know – design shouldn’t be reactive. But it should also be able to integrate the new information that is coming in.

AD: Mmhm. Exactly.

AC: And – I love trans-creation. I mean so much of our lives is trans-creation right? Like –

AD: Absolutely.

AC: in having conversations, in like building relationships. How do we express what we need to say in a way that the recipient can understand it. In order to do that you have to have some empathy, and actively work on that to have a really fruitful and productive dialogue.

AD: Right, exactly.

AC: I love that. Do you have any other examples of how you had to trans-create something in – Argentina. Let’s go to – let’s go to Paco’s home country.

AD: I don’t have any examples that are quite as striking as that one. That whole process was honestly just such an eye-opening experience around – we can have these ideas here in San Francisco that really resonate with us and that feel – we’re so confident that they’re the right thing to do. And then when you start –

AC: Mmhm.

AD: talking to people outside of the office, and again this is the beauty of research. You start talking to people who have not been thinking about this for the last you know week, months, years. Research is a constant trans-creation process. You’re constantly trying to get into someone else’s head and see what you’ve been working on from their perspective. And you realize all the different biases and all the different assumptions that we just make on a day to day basis, as soon as you put something in front of someone who is a relative stranger and you hear their initial reactions and you learn how –

AC: Mmhm.

AD: they’re responding to something.

AC: And I think also, what a great lesson in how much richer our own – not just for professional reasons and creating products, but how much richer our own lives become when we’re open to hearing all of these different things.

AD: Mmhm.

AC: And open to having our own perspectives and biases challenged. Our – I’ve been doing a little bit of soul searching on – Anne what you had raised earlier. And I love how you think about how you go into asking questions without having a hypothesis. That’s so hard to do. Because you’re like well, I think this. I bet I can find 10 examples that prove that –

AD: Mmhm.

AC: Right, like – and if you already have a hypothesis going into a challenge, or going into an information gathering mode, you know obviously that’s going to slant things right? And I’ve – so I’ve been reading about p-hacking in research? Forgive me if I butcher it cause I’m a layman coming to this. A laywoman coming to this. But as I understand it, essentially it’s being – it’s taking a data set and slicing it with different hypotheses until you essentially come up with like data in one hand that fits the hypothesis in another. So that something then gets proven. I sometimes worry that like we’re at – as a culture kind of at danger of doing this at large with all of our assumptions and all of our unconscious biases. I really feel good about the fact that you’re nodding as you’re listening to me say this. And to me again this is just such a reminder about the importance of having different perspectives in a room, to be valuing a diverse set of viewpoints. And to always be thinking about how do we get more information. Always be questioning ourselves and how we look at that information.

It’s been kind of occupying me in my mind especially as like – I think in the past 6 months, year I’ve really had my world view challenged by the political climate. And I would like to know you know is this something that – Anne maybe that you think about or if you any thoughts on it.

AD: Oh yeah. I mean I think we see examples of p-hacking happening constantly. And where we’re presented with data that gives us one vision of the world. You know I’m thinking about the NY Times leading up to the election there was that like – here’s the likelihood that Clinton will win versus Trump. And how I think that –

AC: Mmhm.

AD: amongst my very liberal close personal circle, how that just gave us all so much confidence that this was in the bag and

AC: Mmhm.

AD: no big deal. Or you know more recently there was that survey that the Trump camp put out, asking questions about a whole range of issues that were really written in such a way that that of course the story no matter what people answered was going to tell the narrative that –

AC: Mmhm.

AD: the administration wanted to hear. Yeah I mean I think like data can be very dangerous because we all do have biases and we all do want to hear the things that reinforce our own beliefs. I won’t say that I’m great at doing that all the time. I think you know we’re all human and one of the ways that I’ve been trying in my research to make sure that I’m staying open minded is when I figure out who I want to talk to – I do that based on characteristics that are already proven. So for example, if I’m talking to guests, I come up with criteria of – you know I wanna talk to people who traveled on Airbnb for the first time in the last week. And the reality is, there are so many different people for whom that’s true. And so just by narrowing down my criteria to something again that like I know is true in the data, and I’m not looking for things like where – where do they live and what their gender is and what their age is. I can sort of automatically get this broader, much more interesting set of people who match that one criteria. So that’s one sort of – it’s not even a great strategy really, but that’s one way in which I try to make sure that I’m talking to a wide range of people.

AC: Mmhm.

AD: By narrowing in on what I think is the most important criteria, then letting everything else be open from there.

AC: Open. That’s – that’s definitely a recurring theme in this conversation. How do we be open? Which I think leads me to my final question, which is – how do we do better? You know how do we ask better questions, how do we challenge our own assumptions? How do we bring more perspectives into our work? Paco if you have any tips or any personal ways in which you are tackling this challenge?

PV: I mean I agree with everything that Ann was talking about. I think from my perspective the things that I – that I tend to do and they’re not revolutionary and they’re not – they’re not by any means my invention. But I try to – whenever I come into a design challenge is to ask the why question. And to ask it you know at least 5 times. You know I’m sure this is the Toyota way of manufacturing right? And to really driving to the core problem that you’re solving. We all come with preconceived notions of what the problem is and how to solve it. And usually it can be very different. So you know in the past few years have relied on that system to ask why at least 5 times. And with time you get better at asking the next question, right.

AC: Mmhm.

PV: I found that if you do that in practice, you usually get to a more focused core of the problem. Which can – then when you back out of it you can design a solution that’s much more effective.

AC: I love that. Asking why and asking why continuously throughout the process. Anne, what about you? Any single tip?

AD: Yeah, I’ve been thinking a lot recently about this idea of a growth mindset. I don’t know if you’re familiar with that but it’s a concept that a psychology named Carol Dweck out of Stanford has spent a lot of her career thinking about. And it’s really beautiful concept which is that when you have a growth mindset. So when you’re in – when you approach the world and you approach problems with this idea of I can learn and I can grow. Which also inherently means, I don’t know right now. You absorb information and you can change your mind at an incredible rate. And this is a really powerful concept in children but it’s also been proven out in adults as well. So priming people with this idea of a growth mindset, that you can change the way that you think. You can learn a new skill, you can learn something new – can make people open to so many different experiences and so many different new ideas. And so I’ve been trying to remind myself constantly that I wanna be in this growth mindset as opposed to a fixed mindset which lends itself to saying things like, oh I’m an expert already. Or I’m really smart or I’m a great designer or I’m a great researcher. Constantly telling myself, I can be a better researcher. I can learn more about this. I don’t know as much about this as I want to. I really want to to understand this better. Growth mindset is my – my hack

AC: I love

AD: to that.

AC: that. Always ask why and always have a growth mindset. Well with that, thank you so much Paco and Anne. This was a fantastic conversation.

AD: Thank you.

PV: Thank you.