meriko borogove: designer, engineer, director and photographer

This transcript was generated by an automated service. In some cases it may be incomplete or inaccurate due to inaudible passages or transcription errors.

meriko: [00:00:00] We were literally locked behind a glass door. I remember knowing it was going to be important and knowing it was going to be big. But none of us, I don’t think, really understood. You know, and Steve came and talked to us about it. He dragged us into a room and he said, I’m not here to yell at you. I’m here to tell you to pay attention to what’s happening because you’re changing the world.

And I’m lucky enough to have done it once before. And these are the stories you’re going to tell your grandkids.

evan: Welcome to Design Tandrids.

josh: It’s a podcast from cool hunting, exploring the creative processes and inspirations that drive change makers. I’m Josh Rubin, and I am getting over a cold. So please excuse my particularly nasally voice for this episode.

evan: And I’m Evan Ornstein, Josh, and I’m always excited when we record our podcasts. Uh, but this one in particular, I’m also, you know, I’m really excited about, uh, it’s our friend, Mariko. And not just because she’s one of our favorite people and one of the smartest people we know, but she’s also a dear friend and one of our favorite people to nerd out with.

I think it’s also important for our audience. If you’ve used any Apple devices that have a camera in them, this is the person who brought those to life. It’s a pretty big accomplishment. It’s really challenging for me to summarize everything that she does, but I’m going to have a go at it. Mariko uses creativity and technology to serve people and community and art and togetherness and

josh: experience.

I think that description works perfectly of, you know, we first met Mariko at a mutual friend’s wedding many years back and she was taking a group photo and I made a comment on the angle of the sun and asked if maybe we should move. Someone in the group who I knew worked at Apple said, she’s in charge of our cameras.

I think she knows what she’s doing. I remember that. Yeah, of course she did. And I was [00:02:00] intrigued and we’ve been nerding out ever since all

evan: of us, but especially you two, um, and given your shared experiences as very good photographers, user experience designers, and some of the earliest people involved in creating.

Mobile phones for the world. I think you have a lot to talk

josh: about. Well, that, yeah. And, and like you have, I’m insatiably curious and I always learned so much from America. So with that, let’s, let’s jump in. Welcome to design tangents, America. Hey,

meriko: it’s great to see and hear you all. You know, you two are probably the only people in this world that I would get on a Uh, some sort of podcast with knowing that I’m going to be washed in hot pink and I’m still here.

josh: Yes. I don’t think I’ve ever seen you wear anything besides black.

evan: It’s good to step out. So America, I know that you studied biology and you found your way to design a technology pretty early in your career. Can you take us through that journey a little bit? It

meriko: starts, you know, all the way from. when I was a kid, right?

I was a dreamer. I was an avid reader. I was a theater geek and I was a gamer. Uh, my dad’s a laser physicist and my mom’s a kindergarten teacher. People say that explains a lot about me and I try not to be offended. Um, but you know, I’m also a Gen Xer. I grew up in the eighties. Uh, I’m the first kid and a big sister, which means I have no small ambitions, but I’m also really practical about getting things done.

You remember, this is a good time to talk about Dune. Uh, do you remember the 1984 Dune? Um, I was 11 when that came out. So. I was really into theater, thought maybe I wanted to be an actress, but probably the first woman president of the United States or a scientist, small ambitions with the actress who played Aaliyah, like the kid actress in Dune, who was exactly my age and she was wicked smart.

And she said she wanted to be the first woman president of the United States. And I thought, Oh, I’m never going to win an election against her. [00:04:00] So scientist it was, uh, and went on. on to do that. You know, I, I went to, to go be a biologist. I studied evolutionary developmental biology at UC Santa Cruz, um, and was taking a year off before I went to grad school because I had good GREs, but not good enough to get funding.

And I mean, quite honestly, the baby queer wanted to live in San Francisco for a year. And so I took a temp job at Apple on the Quicktime team in 1995. Um, and it turns out that engineering is not that different than science, especially if you grew up using computers. So

josh: you joined when the company was still fringy and rebellious and then later helps to launch the iPhone, which arguably is the product that took the company big and took the company into the mainstream.

And as someone who I think of as still somewhat fringy and rebellious, I’m curious how, how do you feel about that massive arc?

meriko: I mean, it’s pretty wild. And it’s wild to look back on. When you’re in it, you’re just making work. Um, but yeah, when I started, Gil Emilio was the CEO. So I was there when Steve came back.

I was on the QuickTime team, which was weird and fringy inside of Apple. Because we worked on both Mac and Windows. Um, and you know, I was a Unix kid, which set me up well for the Intel transition. And then Mac OS X and then the iPhone. But it’s interesting because when you’re making creative work, uh, you’re fringy and, you know, it turned out me going to clowning school helped as we were building the iPhone.

Because when you’re Recording slow mo video, juggling is a really good test case. I didn’t

josh: know about clowning school. Um, we’re gonna talk about that later.

meriko: You know, it’s, it’s, uh, interesting, especially when we talk about the phone, because, you know, we didn’t sleep for a good year and a half when we were making the phone.

We were literally locked behind. A glass door. I remember knowing it was going to be important and knowing it was going to be big, but none of us, I don’t think really understood, you know, and Steve came and talked to us about it. He dragged us into a room and he said, [00:06:00] I’m not here to yell at you. I’m here to tell you to pay attention to what’s happening because you’re changing the world.

And I’m lucky enough to have done it once before. And these are the stories you’re going to tell your grandkids. But if you don’t pay attention, you won’t remember them. And it’s funny cause I remember in that moment really thinking like, Oh crap, like, right. Cause you get, you just get really focused on making the work when you’re nerdy and fringy.

I think that’s sort of one of the most fun things about working with nerds. who are really into what they’re working on is that you get, you get really stuck in, but yeah, it’s pretty wild. Um, did

evan: you do that? Did you like journal or how did you make a mental note of some of those events or stories or

meriko: anecdotes?

I for sure journaled. Um, I have a million pictures and a whole lot of movies, um, and a lot of writing because I think as a reader, I’m also a writer. So I have files and files and files of writing about that time, both from that time and since then. But yeah, it’s huge now. Like it’s, it’s, um, it’s unbelievable.

And kind of going, going through the time, one of the things that was really interesting in that is that you have to figure out how to, I mean, the word innovate is now kind of a dirty word, but you have to learn how to make new things, um, in, you know, wild and fringy ways, but the way you do it. It’s different depending on the scale of both the company and the team and the people making the work.

And you know, when I was in my twenties, there were the folks in their thirties and forties who had been at Apple for the 10 years before me who made space for the way I was doing things in the way our team did. And then, you know, 10 years after we launched the first iPhone, I found myself talking about as a leader, how do we make space now that the.

You know, the teams are bigger, the product footprint is bigger, the people who, the number of people who are using it is at a scale that’s really different. And so, you know, figuring out how you make space and hold space for people to do that in different ways, [00:08:00] that scales is a, an interesting problem.

evan: So America, as you’re working on creating the original iPhone, that work encompassed many components, right?

I remember you mentioning previously that included the ringtones. other things, um, including software and some other technical stuff. That’s a little bit above my head. Um, but cameras clearly became the focus of what eventually became the focus, I would say. Yeah.

meriko: Well, I mean, We were such a small team on the first iPhone.

We all did a lot of things, right? I was responsible for power and performance. Um, I was responsible for the QA team. Um, we built all of the demos that Steve launched with. Um, but it was also really embedded and responsible for the media stack. And so that’s graphics, audio, and sound. And because I worked really closely with the human interface design team, I wound up both working on those sounds and editing them and putting them through.

One of the things that I knew when we did the first phone was that we had a system that was powerful enough to be a gaming system. Um, we had a GPU in there that was the same as the PlayStation of the time, a graphics processing unit. It’s the thing that renders all of the graphics on your computers and your phone.

And it’s something that game developers care about a lot. Which

evan: is kind of amazing.

meriko: Yeah. And we had put the software on the phone that was the software that game developers would use to make high end games. And so we looked and said, this is, this is really going to be a gaming device. And we worked on that.

And the other thing I believed having worked on the Intel transition, so understanding that, you know, we could have influence over our hardware and what that meant, but also working on camera raw and aperture before we made the first phone, I understood computational imaging, right? And that, you know, That was really a powerful thing that you could do with photos and software.

And so I believed that we should not be competing with camera phones, which all

josh: sucked. Of [00:10:00] course. Absolutely. Yes. All camera phones

meriko: sucked back then. They were so terrible. And I knew that we could be competing with point and shoots and DSLRs because we could control our hardware. We had this great software stack and we had the ability to work on those things together.

Um, And that was a hard sell during the iPhone 3, you know, the iPhone and the iPhone 3GS and the 3G. We started working with a Canon digital elf. We bought it for everybody and we started on our tiny little camera team, taking all of our pictures against that. And I think it was the iPhone 4. Which was about the time that I had to stop and focus only on the cameras that we really started competing with point and shoots.

I was really proud of that phone, really proud of that camera. And this

evan: is when, for people who don’t know the, all the dates

meriko: on all this. Yeah. The iPhone four, that was 2010. So you’re

josh: describing the relationship between hardware and software. And You were talking about computational imaging. Is that prior to iPhone four?

Was that computational imaging about post processing a photo that was taken or were you already? using software to help the photo that someone takes look better.

meriko: I mean, we were doing a little of that up front. You know, the, the hardware was a single chip that we didn’t build at Apple in the first iPhone.

And it had a lot of problems for the nerds in the audience. Every single piece of landscape video that played back piped through the camera block because the video rotator was broken on the video hardware decoder. Um, so everything ran around and. Piped through the camera and came back out to the display.

Um, and anytime you played a video, but yeah, we were absolutely using software along the way where you’re talking about exposure and exposure compensation and picking that up as early as the iPhone three GS, we were doing, uh, something we called time machine, which was when we started saving off the three photos that happened as you hit the, take the picture button.[00:12:00]

And we would do a quick analysis on which one was the sharpest. And we would save that one. Um, we didn’t have enough memory to hold them all. So you couldn’t have access to all of them, but we were doing that underneath with the iPhone four, when you did tap to expose and tap to focus, there was a lot of, um, computational imaging or computer vision work going on underneath.

That was also the front, the phone that had the first ISP, which is an image signal processor that was designed. It was the first phone that had an Apple designed. chip in it, and it had an image signal processor that was developed just for the camera for the features that we were trying to build.

josh: And was that the keynote when we saw this kind of presentation of several images that were captured at the moment that someone hit the shutter button on the phone?

Yeah, And learned about like we as consumers learned this, this concept of computational photography where you hit the button once, but a bunch of photos are taken. They’re all analyzed for different variables and then merge to make the best possible image. Was that the iPhone four?

meriko: No, that was the iPhone five S we didn’t really start telling that story for a while.

josh: That story blew my mind. That was, that was just a really, uh, incredible moment. And, uh, you know, just a very exciting, impressive time for me as a photographer who always still carried a, uh, another camera. Like yes, the iPhone was a convenient camera that was always available, but often I had another camera with me and that was maybe the beginning of when.

I started thinking, Eh, maybe right now I don’t need another camera. I can just take my iPhone. We’ve talked

evan: a lot about spatial computing and looking at ways, the way so much people engage with it. And clearly cameras are part of that. Um, I know there’s a lot of, uh, Secrets and things you can’t share, but we, we kind of assume that you worked on the vision pro and we know that you have one and that you [00:14:00] like using it.

Um, what are your thoughts for this new way that people can really immerse themselves in experience and interactivity?

meriko: It’s important in a couple of ways. The first one is, is, you know, inherent in that word spatial, which is if you take a kid today, they’ll walk up to a TV and try and swipe on the TV and not understand why they can’t, you know, pinch and zoom on the TV or move it around.

I mean, I firmly believe that in not very many years, kids are going to look at photos and not understand why they can’t look around something that’s in the photo or in a movie. and have some agency about what they’re looking at. You know, I think the Vision Pro is a game changing device. As you know, I’ve been working in VR since before I left Apple and looking at both VR and AR, but the Vision Pro I think is the first time.

And I’m, I’m so proud of the team. I think that they’ve done an incredible piece of work here, but you can move around. While you’re in it, you can see things that you can’t touch, which is pretty interesting. You start being able to imagine things, but because we are like mobile, social creatures, this ability to work in a spatial way and to be in a place in an embodied way.

is compelling. Um, and I think that it’s the first time, I mean, I’ve been taking meetings in walk about mini golf for the last three and a half, four years. Um, you know, first in a quest one, and then in a quest two, and now in a quest three. And it’s really, it’s really impressive. Like if you’re talking about being together when you’re apart, because we are embodied humans.

You recognize how people move. Um, when you talk about the psychology of attention and when you talk about the psychology of perception, um, That you are not a flat thing in a window in a screen, [00:16:00] but that you are moving around in the same way that your head moves, and that we’re talking, and that the physics of the situation that my body is using, it drops you into being really present.

And so this is the beginning of how we foster presence and connection in a spatial computing, you know, in an augmented way, I think.

josh: Miracle, I love what you’re saying. saying about agency and about this idea that, you know, now kids try to swipe a TV screen or a magazine. I’ve seen that happen too. And eventually they’re going to try to walk around a photo where they’re going to try to look at an object that is on a screen or in a screen and try to move around it.

And, you know, that, that is the idea. a potential influence of the future of spatial computing. Um, I want to come back to just of like one last little bit of kind of history, um, thinking about the significant leap that happened around iPhone five and this, this whole conversation around computational photography and software and hardware working in concert to create the best possible image.

I have to imagine that was. An exciting time. It was a potentially challenging time. Uh, you know, there’s, there, there was a lot going on. Um, we’re going to throw out a break, but when we come back, you know, if you have any stories from that moment in time, we’d love to hear them.

evan: And, and before we go to break, just, you didn’t really answer whether you worked on vision pro.

Or not. Not sure if that’s a wink, wink, nod, nod, or Well,

meriko: I mean, I did. I did. Okay,

evan: great. Now we can go to break.

josh: So, do you have any fun little stories to tell us?

meriko: Well, you know, the, the first one is, um, it’s, it’s all about consent. [00:18:00] When you’re the person that everybody wants to show you a photo, you learn to never, ever swipe in someone’s camera without permission.

josh: Yeah. Yeah. Yeah. Yeah.

meriko: Uh huh. Um, consent is sexy. Other people’s camera rolls are not.

evan: Uh,

josh: Wait. That, that’s a t shirt. That’s a t shirt.

meriko: Um,

evan: Did you maybe do that and see something? that we should all know

josh: about. I think that’s America’s point, Evan. I feel like I just told you. Yeah, I feel like she just told us.

evan: You didn’t say whether you were swiping or someone was swiping your device.

meriko: Oh no, when you run the camera team everybody wants to show you a photo.

On their device. Either a photo that was really important to them that they have because they had their phone or a photo that didn’t work or has a problem. But everybody is always handing you their phone to show you a photo. I don’t, I don’t hand other people my phones. I worked at Apple.

evan: You are no longer at Apple. You did leave a couple of years ago and you kind of moved on to doing new things. And I think an important stepping stone to getting you there was clowning school. So maybe we can jump back to that and see how that’s, you influenced this next chapter of

josh: yours. That’s well done.

Good transition there. Yeah. Way to bring back clowning school. I have clown nightmares, but I’m, but, but I, I’m, I’m here. I was a

meriko: pretty terrifying clown. You know, we used to do, uh, what we’d call the stupid movie for the graphics and media state of the union, which was the talk that we would give right after Steve’s keynote at the developer conference every year.

And so we would make a movie that would use whatever technologies we were announcing. And we always did it wildly underslept, but I think it was the The year after we shipped the first iPhone, I actually wound up having to be a mime in a movie that [00:20:00] was about Web 3. And I was a wrong mime type. Josh is going to kill me.

josh: Yeah. Yeah, I might. I don’t know if I need to see that. I might need to see that. Yeah, I need to see that. I need

evan: to see that. I

meriko: totally need to see that. Well, we can, we can, we can, uh. The

josh: wrong mime type. Love it.

evan: Um, so since leaving Apple, you’ve started this whole new chapter and I know it involves theater and live experiences and academics.

Um, and just thinking a lot about where the, this part of the world is going or the way in which we interact with the world is going. Maybe you can share a little bit about kind of what you’ve been up to and what you’re thinking about these

meriko: days. You know, it’s, I feel really lucky to have been pretty cross disciplinary myself growing up and then to work for the first 20 years of my career, creating the, the tools and the, the tools.

and the technologies that people build creative work on. Um, over the last six or seven years as I’ve left Apple, I talk a lot about how I’m still working on those technologies and that research, but from the point of view of making the work. Um, and it’s a great pleasure to shift and get to, to focus on that.

So, you know, I work with theaters, I work with, um, studios, both art and technology studios. I’m a honorary professor at the Royal Holloway College of London. Um, and I’m working with them on defining research for excellence in screen and performance technology over the next five or six years. Um, I got to, I got to co direct a play, uh, at the Berliner Ensemble last summer.

Um, that was using a lot of spatial technology in the service of the [00:22:00]stage and in the service of how to take a story into multiple places at once. Um, and I think it’s, uh, a lot of it comes back to holding space for what’s possible and being able to imagine what’s possible. I think what drove my work at Apple, um, you know, my most recent lover, wasn’t the first person to say that I reminded them of case Pollard, which should resonate for you all, but like,

josh: William, William Gibson reference pattern recognition is the book.

Second time that, that. That reference has come up in our world in the last few months and hadn’t in several years. Sorry to interrupt, but just calling out the reference.

meriko: Yeah. Um, well, we’re going to have to talk when Apple releases the new Neuromancer.

josh: I know. I know. I’m very excited for the new Neuromancer.

meriko: Um, but this isn’t a podcast about William Gibson. It could be.

evan: Um, I want to call out one thing because I remember you were so excited about it and talked about it a lot and we participated in it. And that was, I believe, sometime in like spring of, Of 2021, it’s the Royal Shakespeare Company’s, I think it was called A Midsummer Night’s Dream.

And it was an interactive performance that was done online. So guests could watch this performance online, which isn’t unusual in and of itself, but that there were interactive elements as part of it. That was

meriko: a big demonstrator piece for a program called Audience of the Future. Um, I was in the advisory group for it.

It was led by the Royal Shakespeare Company, but it included the London Philharmonia, uh, Marshmallow Laser Feast and a handful of other people. And it was a live play set in the, on Midsummer’s Eve in the world of A Midsummer Night’s Dream. It was done with live actors who were in motion capture suits being captured Both spatially and their movements and recorded with cameras as well.

And then they were piped into, [00:24:00] uh, an Unreal Engine game engine set, which is how we made the interactive play that was happening live. And what that means is if you’ve heard of Fortnite, um, that’s the same engine that’s used to make Fortnite or the concerts that happen in Fortnite. And it was this live performance that was seen by over 80, 000 people over 10 Performances sort of deep during lockdown in the pandemic.

And it was stringing together live theater and gaming and technology. And. interactivity so that you had a sense of presence and connection both from the audience to the actors, but also the audience members with each other, which is an important part of any sort of live performance, whether it’s in theater or in music.

Um, Sarah Ellis, who’s the digital director at the Royal Shakespeare Company, um, who brought me into this talks a lot about jeopardy as well, which is really fundamental to theater. And I think it’s really interesting when you talk about any kind of spatial media and that’s the sense when you’re in a live event.

That anything can happen and you’re in there together and something could change. Right? That’s not something that you have when you watch a movie, when you go to an IMAX theater and it’s a, you know, well edited recorded movie, or when you’re listening to this podcast, we all know that we can edit the podcast, but when you’re listening to a live broadcast or watching a live piece of theater, there’s this sense of jeopardy, which is a thing that really develops in.

So that was a really interesting and challenging and special piece of work because it tied together so many things at the edge of what was possible and deployed it to so many people because we deployed it in a web browser instead of in a VR headset. It was a really big deal. It was quite a thing, you know, and what was interesting is that it was a pivot.

So that was a piece that they were supposed to put on in June of 2020 that was with Magic Leap. Glasses, um, augmented reality glasses that was going to happen, um, around Stratford on Avon at the same time that the play was kicking off the season. And so there was a whole sort of live action thing that was supposed to happen.

And then, well, we all shut down and they had to pivot and think about what, what would they do to think about who, who [00:26:00] are the audiences of the future and how do we reach them and how do we serve them and where it had been, how did the audiences in the future be together in person. We had to turn to how do audiences in the future be together when you’re apart?

josh: It came together so beautifully and people people were together there was Engagement there were some, you know, some people just watched Uh, but even though it was on screens or in headsets, it had the, that sense of jeopardy that you’re talking about because we, uh, because, because it was made clear that this was happening live, just gave it a different level of excitement.

You know, something that you mentioned to us a long time ago, that’s always stuck with me is that one of the things you do, or one of the ways you look at what you do is designing stuff that people didn’t know they wanted. And that, that work being kind of on this edge of fiction and reality and what we’re just talking about with audience of the future is an element to that.

Uh, I think a lot of, a lot of, uh, the other work that you’ve been doing is an element of that. And presumably, you know, You’re a sci fi fan, just a hunch, um, and, you know, and, and sci fi is, is definitely an inspiration for the creative mind. Is there, is there a point when you flip from simply being entertained to analyzing the The narratives of, of sci fi for signals of the next great thing.

Are you, or are you, are you always analyzing?

meriko: I was going to say that there’s not a point where it flips. I think it, it’s more that I don’t know how to turn it off. That whole pattern recognition, pattern matching thing. And the imagining that that’s just, that’s just how my brain functions. Um, you, you want me on your continuity.

If you’re making a film or writing a script, [00:28:00] um, because I’ll immediately spot all the problems that made me really good at QA as it turns out. But I think, yeah, I don’t, I don’t know how to turn that off. It’s all, it’s all integral and integrated. Like you dream about it and then you think about how to make it all at the same time.

And like the worst science fiction is the stuff that over explains things that are already not true. which is some of the lesson around making things and designing things that people don’t know they want yet, right? The best science fiction leaves space for you to fill in the blanks. The technical details yourself, um, so that they’re as correct as you can make them in your brain instead of, uh, taking them apart.

And they’re about humans, right? And they’re about humans and how they, how we come together. So, um, I think much like what I see in spatial media right now from an aesthetic standpoint, Um, personally, I think a lot of the things that are happening in volumetric capture, where you record someone in 3D, we don’t quite have the fidelity and you hear people talk about the uncanny valley when you see a recording of someone that’s computer generated and they just look a little bit wrong and it kind of creeps you out a little bit and it’s pretty well studied.

But if you back off of being absolutely going for a photographic reproduction, it’s You leave enough space for people to imagine and fill in the blanks. And that’s, that’s how, that’s how our perception systems work in the first place. Um, you don’t actually see everything you’re looking at. You see a bunch of the things in between and your brain fills in the, the parts.

josh: Was that the cats? That’s the cats. Oh,

evan: ready for dinner. Erika. Evan. Star Wars or Star

meriko: Trek? Darling, you don’t, you know, I don’t discriminate based on gender preference and gender identity. Um, yes, uh, it’s a yes and situation for me.

josh: Well, but to your, to, to [00:30:00] the point that you were just making about how much to explain.

And how much to let people fill in the gaps. Star Wars and Star Trek are so different in, you know, in the gaps versus, you know, the, the, the explanations. I don’t know, I personally, I’m, I’m, I gravitate more towards Star Trek. I like, I like the lessons. I like what I get to learn from Star Trek.

meriko: Yeah, I mean, I grew up watching both.

Like avidly, we watched Star Trek as a family. We went to the opening of every Star Wars movie. Yeah. Um, you know, we built the models of the spaceships from both, uh, you know, when I was in college and we went to Disneyland, we ran around singing the Ewok song, a little embarrassing. And those are great. But you know, I think that there’s.

You know, it’s interesting. I’ve been reading a lot, but like, let’s talk about the first seasons of Russian Doll or Orphan Black. Those are perfect pieces of science fiction as well. I just finished this book, A Half Built Garden by Ruthanna Emmes, and it’s fantastic. Everyone should read it, especially you two.

It’s a sort of hopeful post apocalyptic with aliens and sort of hope for recovery from climate change and how we, you I was really sad when the book was over. I finished it last week. You mentioned

evan: earlier that you’re working on lots of different things, and I know that you were just at South by Southwest and you were there with scan lab, scan labs, a company that’s based in London.

And I think in the simplest terms, the work with technology. To create really interesting content, um, and maybe you can tell us a little bit about what they’re doing and why it’s one of the things you want to work on

meriko: scan lab is a beautiful studio. It’s both an art and a technology studio out of Bessemer green in London.

Um, we believe that 3D. is the future of photography and cinematography. They’ve been around for about 15 years and they have, uh, the two co founders have training as architects. And so they encountered [00:32:00] LiDAR scanners in the service of that. And as designers immediately kind of turned to them and was like, well, we should be able to scan all the things people say we can’t scan.

And then they found art. And so they started this studio and I did a project with them when I was still at Apple. And when I left in 2017, I went to visit and I think in 2018, like since 2018, I’ve been, uh, Part time part of the studio. And so what I would say is their media is the point cloud And a point cloud is a a set of 3d points that describes a space and it works by shooting a laser the laser bounces off of something and it comes back and you can get this really Beautiful precise map and the cool thing is before I left Apple we put some Scanning cameras into iPhones and how the face recognition software works, but there’s also one on the back of the back of the camera as well.

Um, and so scan lab works to make art that is in the service of bearing witness and telling stories. Sometimes that’s, um, in film. And in television with the BBC, sometimes that’s making an artwork that’s a novel form of 3D time lapse cinematography about our relationship to the planet. That’s a piece called Frame Rate that’s traveling and touring right now.

I think you all saw it in its special preview at South by Southwest in 2022. We also, uh, when I said we worked on the, I worked on the play at the Berliner Ensemble, that was a scan lab piece where we did all of the sets in the creative direction. Um, and we projected those point clouds onto four scrim in the shape of a box, which created the set throughout, not unlike what you saw at the sphere, but much smaller, um, and you were outside of the box instead of inside of the box.

Mm hmm. And then, you know, we also work inside of headsets. So the piece we were showing at South by Southwest was a documentary about two speleologists, um, people who explore caves for a living, um, done in point clouds inside of a quest to headset

josh: because of you and, and, [00:34:00] uh, learning about LIDAR and point clouds and scanning apps that are available.

I, you know, I’ve, I’ve started using different scanning apps on my phone and have for a little while to capture objects and to capture spaces. Uh, one of them that I’ve been using a lot lately is poly cam and it’s great because I can open those scans in vision pro and walk around the object or exist in the space.

And I’m curious. you know, again, going back to this, this, um, agency that we were talking about earlier, you know, is it, I feel like it’s total nerd behavior when I do this stuff. Do you, do you see intentional creative use of LIDAR going mainstream?

meriko: Yeah, absolutely. I just think people aren’t going to know to call it LIDAR and they shouldn’t need to in the future.

Well, right now, all of the things, all of the cameras, around you in the world that are recording all the time, whether they’re on a, an autonomous car or on a Roomba or, you know, on a street light camera, those are all taking depth data and recording them. Um, and it doesn’t always have to be LIDAR. Um, if you’re looking at computational photography, the technology to take a whole bunch of 2d pictures and use computer vision to recreate the 3d scene.

In an accurate way is happening today already. I will refrain from saying all of the acronyms that Evan will ask me to explain, but I guess the point of it is, is that I think the underlying technology right now, it’s right now it’s nerd behavior. I think in 10 years, that’s what a photograph or a movie is going to look like.

And you’re going to expect. when you take one, that you can do that. [00:36:00]

evan: Thank you for setting up. My throat a break because the future of all these things and the way they come together, the future of entertainment is something that we talk about and dream about all the time. And I’d love for us to dig into that when we come back.

josh: So, Mariko, our last episode of Design Tangents was with Willie Williams. Who is the co founder of treatment studio and the creative director for, uh, for you to for many, many years and recently focused on their residency at the sphere in Las Vegas. And Evan and I have seen the show. You’ve seen the show.

We love that it’s immersive, but it’s not solitary and it’s a mind blowing shared experience. With the advent of that construct of screens and speakers that makes the interior experience of the sphere and real time compute, do you think that the sphere’s approach to immersive experience is the future of entertainment?

There’s a lot of, I feel like there’s been a lot of debate and discussion around, you know, what, like, is the sphere a gimmick? Is it a one off? Is it the future? You know, what are, what are, what are your thoughts? Especially as we, you know, as we’ve been talking about spatial compute and, and experiencing it in a headset versus experiencing it in a space and talking about togetherness.

Versus solitary screen based experiences. I just love to hear on the other side of, of seeing that show, what some of your thoughts are on that format for immersive performance.

meriko: I think it is a part of the future of live entertainment. I mean, it’s a, it’s a new kind of [00:38:00] set and a new kind of stage. I didn’t see the film.

That is, you know, essentially a giant film without a live performer. But this sphere isn’t the only giant LED, you know, spherical compute with really incredible sound. And I mean, the other thing is they’ve got scent and things that they can push through the screens as well. There’s a company called Cosm that is building some domes as well.

And I think that, you know, we’re social creatures at our hearts. Even even us introverts is a community after the last four years. We’re hungry for that. In a way, um, I think that there’s any number of things that, that, that signals for it might be projection. It might be like, hopefully it’s not all 18, 000 people cause that’s too many people for me.

Like it’s pretty cool in that space. It was a lot of people, but the sense that you can create a space that is different than the one that you’re in. that is believable, that is also fanciful, right? Because I think that’s one of the things that was really brilliant about the show that they put together for you too, is that there were times where What was on the screen absolutely felt like physical real set.

Sometimes it made the set feel like a different shape than it really was, right? They were using all of the tricks and, and trade, you know, um, Sarah, Sarah and I, Sarah Ellis, and I talk a lot about how theater is technical, right? Um, and this is initially how we became friends because engineering is, I believe a creative practice and design.

Um, and theater are very technical practices. And when you understand that and work together across disciplines, you wind up creating things that are like what we saw at the sphere, that people can have these incredible experiences that you see together. I think that’s possible in headset today. We talked a little bit about [00:40:00] playing mini golf together a couple of days ago.

I was playing with our friend Carl, who was in New Zealand and our friend Scott, who was in Minneapolis. We played 18 holes. of really wild mini golf and forgot that we were in headsets. And then we sat down in our vision pros and we played, well, they taught me to play hearts. I’d never played hearts before.

A very different kind of experience. And you know, there’s something on your head, but I walked out of that couple of hours feeling like I had really hung out with my people in a way that, and this is, this is great. You’re my friends, but like seeing you in squares, I In a window where I’m sort of tethered to what I’m doing and I’m not moving my body in a natural way, or like we would if we were hanging out and having dinner or going for a walk.

Um, I think there’s also a thing if we’re going to talk about the future of entertainment that is less about the stage and the kind of media that you put on the stage, because I think that that those things are clear, but it’s also about. Agency and conversation and contribution. The more I talk to the young people in my life and, and I’m thinking about how gaming comes into theater, or when you talk about immersive theater, which is a whole lot like interactive fiction and the Infogom games I grew up playing, but with actors, they expect to be contributing.

To what’s happening to be a piece of the work as well, not just witnessing the work and that kind of comes to that to jeopardy when we were talking about that shared understanding and belief that something could change and I think there’s something in the future of entertainment that is both spatial that relies on being able to put things into the world that you know, we can see but not touch or that we can create and that we can have that shared experience whether you’re in a stadium together or Um, we’re here together and I feel like you’re here with me and I believe that you’re here with me.

Um, but there’s a part of that is that we’re creating it together and we’re contributing to the thing that is the [00:42:00] entertainment. And I, I don’t know quite how to describe it yet cause it’s kind of that thing over the mountain that I think is coming that is enabled by all of the things we do at the sphere, all of the things that you see theater doing when you talk about maybe sleep no more in New York or the burnt city in London from punch drunk.

There’s something going on there that is the future of both education and entertainment, and where those cross over in this, in this spatial space.

evan: Listen, there’s always been change and evolution in all these categories, right? Whether it’s education, whether it’s entertainment, but I kind of feel like we’re at a point where there’s so many new skills you have to have and an understanding of new technologies that you have to have.

And of course, you Experience and history of what that body of work has been in the past to move it forward in my right in feeling like there’s got to be a new generation of people coming in to all of these areas to really think about the world and the kinds of entertainment that we want to invent.

It’s not just evolving because a lot of this is invention. Do you think that’s right? Or do you think we can just rely on, you know, the, the group of people who are already doing those things to kind of move us forward?

meriko: No, I mean, I think you’re absolutely right. We’re inventing not just the first new user interface in the last, you know, of the last 15 years right now.

We’re inventing and discovering because some of it is discovery, right? The next generation of a media language in the same way you have a language of photography that’s pretty well understood about, you know, what a, what an aperture setting or what a rule of thirds has, has you feel when you compose something or black and white or rules of like, there’s a language of cinematography, right?

A Dolly Rack poll makes people feel. angsty and twitchy. We figured that out with Jaws. That’s what happens in Severance when you close the elevator door and that everything goes a little weird. It’s a micro version of that. But we haven’t yet discovered what all of those languages [00:44:00] are in this space. And it does require a lot of new kinds of knowledge and it reminds me a little bit of like You know, early web days where a lot of stuff was really scrappy and people were coding things by hand and hacking around and doing stuff.

And, you know, it’s, um, really multidisciplinary and it’s people who are playing and trying and, you know, I’d say for folks of our generation, one of the, one of our, our roles here is to bring through. Another generation who is figuring out how to do it in their way. In ways that think about things in a really different way than we might.

I know my brain works really differently than my dad’s, right? He’s an engineer, but like he had a slide rule and logarithm tables. And I grew up with a computer in my house. which set me up to do what I did 25 years ago. I believe there’s absolutely a generation that is going to be dreaming past us and building past us.

And I believe it’s going to be multidisciplinary. I think it’s going to be founded in a place where people start from a, uh, a spot of respect, which they think is a thing I learned at Apple to give some credit where that is due is that, you know, you assumed that everybody was there because they had something that they were an expert in their field and you didn’t talk down to anybody, no matter what their role was.

And they didn’t talk down to you. And that enables you to get past so much and to be able to invent new things when you hold that space. And that space is, you know, um, in, in psychology, they call it, you know, psychological safety. In the theater, they call it creating the circle and holding the circle, um, for people to be vulnerable.

Um, and I think that all the young people that I talked to Understand this in a really innate way that at least in the tech world, most people haven’t really understood for a long time.

evan: I think it’s really cool.

josh: As we see every iteration of, of user interface and user experience and the different ways that humans [00:46:00] interact with computers, there’s one layer where we need to constantly learn these new languages.

And there is also the underpinnings of kind of cognitive science and psychology that I think almost always apply. And I’m wondering, you know, if you’re, if, if you are giving advice to students, to parents, what kind of skills should, should people be focusing on, you know, is it, should, should they be focusing on the current technology, the most current technology, maybe a little bit, and maybe also focusing on, you know, some of the, some, some of the underlying disciplines that might be helpful.

That was a terribly worded question.

evan: Yeah. It’s like, should people be studying Greek? Is that going to help them? You know, is it history? Is it math?

meriko: I think all those things are important. Um, and I think depending on like when I give advice or, you know, when I teach or when I go to, uh, to schools and whatnot, you know, I went to UC Santa Cruz and we didn’t have grades.

My transcript is like an inch and a half thick because we had things called narrative evaluations.

josh: I, did we ever bond over this? Cause I went to Hampshire college where we didn’t have grades and we also had narrative evaluations. No.

meriko: Oh, look at that. Oh, well look at that. Um, and you know, I was going to be a scientist, but I intentionally went to Kresge college, which was one of the literature colleges and also the biggest hippie of Hippie college of all of the ones at UC Santa Cruz.

And I did it because I, I mean, I believe that the things you learn, like we are creatures who are motivated by each [00:48:00] other and by story and how we tell stories and how we relate to each other and how we build the things that we build. figure out which things you enjoy building with what tools are available to you and which ones make sense for your brain and go from there.

But like, don’t, don’t forget to be curious and follow the threads of what you’re curious about. And I think those things serve all kinds of folks. I mean, when I was growing up, we didn’t really, uh, have words for neurodiversity, um, and neurotypes and how people think and work. And I think there’s a lot more understanding of that now, which is pretty cool.

Um, but yeah, yeah. Make sure that you read, make sure that you make music, make sure that you go outside and touch the ground and see the sky, walk into some trees cause you’re thinking too hard. And then like, dig into the tools that make sense to you, that feel good to you to use. And then go find people who use the tools that you don’t like to use and invite them to work with you.

evan: Given all of the things that you just talked about, Is that how you’re thinking about the future of entertainment? And is that what’s motivating you? Or have you already kind of leapfrogged? Are you, are you

meriko: developing that? I like to think I’m taking my own advice.

One, you know, one of the things that, that I do that is, I’m really good at what I call squinting through a prototype, which is being able to see the possibility of what something can be. And, you know, this is a thing that, that I learned from Johnny, I have in the design studio, which is that there are times when ideas are really small and tender and you have to protect them.

And it’s really easy to, to, to kill them. And there’s a bunch of times where you do have to stop and say no, but you need to give ideas, the space and [00:50:00]time to grow. And I think one of the, you know, one of the great luxuries of my post Apple career and job is the time to really feed my brain, to put my head inside of everything, to put my body inside of all of the things, to, you know, read and to travel and to then talk and collaborate with people in these disciplines that I didn’t have time to really dig into.

Like I learned so much at the Berlin Ensemble and with the Komische Opera last summer doing very physical things, um, both in how to work, but also in what we, what we can make going forward. And I’ve just talked myself down a rabbit hole. I have forgotten where I’ve gone. That’s not a bad thing.

evan: Yeah, that’s the space that you, that enables you to create,

meriko: right?

Yeah. You know, our friend, our mutual friend, Duncan, who is one of the most brilliant designers I have ever met. Um, when I was stressed out after I left Apple, he said, don’t, don’t get too worked up about it. You’re setting the condition. Like when you’re in a design process, there’s the, there’s the portion of time where you’re setting the condition and you’re feeding, feeding the ideas, um, and putting that together.

Yeah. And you don’t have to name it right now, you know, you can, that’s, that’s part of the process. And so I think, yeah, I mean, I don’t know that I have the hubris to say that I’ve leapfrogged everyone. Um, but I like to think I have some good

josh: ideas. I’m always inspired by your ideas. Just saying.

meriko: I’m always inspired when we are talking about our ideas.

It’s one of my favorite things.

evan: Mariko, thank you so much for nerding out with us. I think it’s the best way. This is the nerding out episode for sure. We’re really grateful for you. Taking the time to share your thoughts and most excitingly for me, you kind of your visions for the future of entertainment and how technology is going to be a [00:52:00] big component of that, of that evolution.

josh: Mirko, I always love our conversations and to get to have a conversation. That other people can listen to is a fun twist on our usual nerding out. So thank you.

evan: Maybe we can have a really big mass, you know, play of miniature golf online or something.

meriko: That would be so much fun.

evan: If you’re new to Coolhunting, we’ve been reporting on creativity and innovation since 2003.

Visit coolhunting. com or coolhunting on social to see what we think is worth

josh: sharing. If you enjoy Design Tangents, please follow us wherever you listen to your podcasts, and every positive review you leave is very much appreciated.

evan: We’d also like to thank Master Dynamic, the official headphones of the Surround network, and our original music was created by our friend Matthew

josh: Deer.

Design Tangents is produced by Surround, part of the Sandow Design Group. Discover more shows from Surround at Surround Podcasts, that’s podcasts with an s, This episode was produced and edited by Rob Schulte.

evan: Transcripts, show notes, and links for each episode of design tangents can be found on our show page at surroundpodcasts.

com and on coolhunting. com.