Omar Rizwan on shaping computers into friendlier forms

Omar Rizwan is a researcher and developer​​ interested in new computer interfaces and new ways of programming. He previously worked at Stripe, Khan Academy, and Dynamicland, where he worked on projects such as Geokit. He’s also a prolific creator of paradigm-challenging projects such as Screenotate, Horrifying PDF experiments, Hijack Your Feed, and many others.

DEVON: Hello, I'm Devon and you're listening to Tools & Craft, a series of conversations with the designers, engineers, and inventors who are shaping computing as we know it. Today, I'm talking with Omar Rizwan. Omar spends his time thinking about, writing about and building new computer interfaces and ways of programming. He's worked at Bret Victor's lab, Dynamicland, Stripe and Khan Academy. He's also the creator of Screenotate, a tool that captures the text and origin of your screenshots to your file system. Omar also created TabFS, a browser extension that mounts your browser tabs as a file system on your computer. I'm also very lucky to call Omar one of my closest friends. In the years since we met in college, the daily drip of Omar isms in my group chats has been a major influence on the way I think about computers, tools and beyond. Omar, thank you so much for taking the time to chat. I've been really looking forward to this.

OMAR: Thanks, Devon, it's really fun to be here.

DEVON: So what's an example of a computer interaction that you think would be better with haptic feedback?

OMAR: On your computer, on your laptop, or on your phone, there are a lot of objects. There are windows and there are tabs and there are files. I feel like often I want the feeling of, I just want to pick up this window and move it around and feel where it is. I don't know if that counts as haptic feedback in the narrow sense of there's a vibration motor in your phone and having it vibrate when you do something. But I think that sense of being able to actually touch things in the computer and use the sense that isn't just your eyes, that's something that I find myself wanting a lot.

DEVON: Why is that important?

OMAR: Often the way I think about computing is more like, there's some feeling that I have that I want at this gut level. The ability to move things around by feel. We've been using our fingers and hands since we were apes and I feel like there are a lot of subtle things about them that you don't get when you're just kind of manipulating a touch screen or a mouse and using your eyes to point things out.

DEVON: Yeah. It seems like a lot of computer and software design is done by what's sort of the optimal way to do something or some very measurable thing, which is very useful. Measuring things is very helpful, but it can sort of discount the day-to-day feel and just what mood does this put you in, what kinds of things are you more likely to do.

OMAR: It also comes out of a particular history and particular technological constraints of, well, we can make a mouse because we can make the sensors to make a mouse. We can put the two potentiometers inside the mouse so you can detect where it's moving on the X and Y axis. It's like it's easy to make a mouse and it's maybe hard to make something that's really haptically rich, and so there's this technical constraint running us from doing it. Also, I wonder if there are kind of cultural constraints with the way our culture thinks about what does it mean to interact with the world. If we come out of a different cultural background, would they have thought different things were important? What if you had an Islamic computer where there's some prohibition on displaying pictures of people, pictures of animals? So it becomes this completely vector display where you can only draw kind of abstract shapes. What would that look like?

That sort of positing an alternate history. And then the other example I think about is in the 1980s. In the 1980s, we had a bunch of personal computers here in the US — like the Apple II and IBM PC. And in Japan, there was a completely different set of dominant personal computers from Japanese companies, and those computers tended to have much more developed graphics, because of the way that the Japanese writing system works, you need better graphics to be able to do basic things. Whereas in the USm because we just have an alphabet, you don't. We don't really need that rich graphical system, so that's only developed later for us.

DEVON: This reminds me a little bit of a recent essay that you wrote. But there was a footnote in it about how the speed at which a printer prints a piece of paper really changes how you feel about it and how you end up using it. You pointed out that Polaroids and receipt printers have a certain lightness and Post-it note to them compared to a laser printer that might take 30 seconds to print the same document. What changes when you feel like you can print things really quickly?

OMAR: When you feel like you can print things really quickly, it becomes something that you do in the course of you working on something. While you're working on the thing, you're like, "I'm going to print out this little thing because it'll only take a second." You kind of have these ephemeral objects and they also feel ... You can also have it closer to you if it's a smaller printer. It becomes part of your working process in a way that I think a laser printer does not. I think that kind of makes sense too. Because if you think about what a laser printer optimized for, it's like it's really optimized for sitting in the middle of an office and printing out these long multi-page documents. That's actually a different use case from a lot of what I'm interested in with computing.

DEVON: Inspired by a lot of the things you've talked about, I bought a computer for my office. But I find that I've been just papering my walls with different things ever since then. It is different, I could print something at the FedEx near my house. 

OMAR: Right. That's like the extreme version of that. Is, you can always just batch everything and go to FedEx once a week and print it. But you're not going to print the same things if that's your workflow. Your screen is a limited space and if you have a printer and if you can print things, you now have a much expanded space effectively that you can work in because you can have everything kind of spread around your desk. If you have something that's printed out, you now have all the tools of the physical world that you can apply to it. This is something we talked about at Dynamicland a lot. Is that now I can mark it up and put Post-it notes on it and maybe even cut it up and make collages or juxtapose it with other things. Those things are all fairly hard to do on a computer. It's very awkward when you have 50% of your work in physical stuff and 50% of your work in digital stuff, because crossing that boundary is relatively hard.

Whether that means printing things or scanning things or whatever. This is part of the long range vision of what you would want to do with a system like that. Where if you have objects on your computer, you can have holographic projected versions of them on your desk and then you can suck them into physical objects if you want to physicalize them. Or you can turn them back into holographic objects if you have the physical object. This is an idea I've been ... Because the computer exists in the physical space, it's a lot easier to just cross the digital physical boundary just by pointing at things in a space where they are instead of needing to have a computing device.

DEVON: The space of input, output devices that exist is kind of artificially constrained somehow. There's just not that many things that have at least hit mainstream. There's probably a lot of different types of input devices and output devices, but people just don't really use them.

OMAR: This is a really interesting area. A few months ago, I was just looking at the PlayStation 5. They have this new haptic controller. It's like a normal game controller with the buttons and the joysticks and stuff, but number one, it has a really rich, precise haptic motor. Kind of like the Nintendo Switch has one of these too, and the iPhone also has I think a pretty good one. The games can generate really fine vibrations to indicate if you're walking over something that's vibrating or if you got shot or whatever. You get this really fine feedback with a lot of precision and a lot of variation. The shoulder buttons, like the L and R buttons basically have these motors in them, so the game can actually make you feel different things as you're pressing those shoulder buttons. You can feel like you're pressing on a spring, or you can feel like you're pressing on something that has two stops in it.So you can see you get this really fine feedback with a lot of precision and a lot of variation. And the shoulder buttons. The L&R buttons basically have these motors in them so you can actually feel so the game can actually make you feel different things as you're pressing those shoulder buttons, you can feel like you're pressing on a spring, or you can or you can feel like you're pressing on something that has two stops in it, so your finger goes down, and then it stops and you have to press harder to get the other stop. Because the motor can be programmed the motors and L&R buttons can be programmed to resist in different ways.

DEVON: I think this is a really big part of why the iPhone feels so good to people, is because Apple really spends a lot of time thinking about all of those interactions and trying to physicalize it as much as possible. One of my favorite interactions is, on the left-hand side of the iPhone, there's a little toggle. There's a physical hardware toggle. When you have all the sounds on and you toggle it on and off, it makes this click sound. There must be some haptic field going on as well because it feels like I'm cocking a gun or something like that. It's so satisfying this heavy metal feeling. I think that that takes a ton of work to do consistently throughout an entire interface. It feels sort of superficial, who cares how it feels? The whole point is that it just locks. As long as it's functional, why do you need it?

OMAR: Right. I also feel like if an interaction like that is satisfying, that actually gives you a different relationship with it. I kind of want to make more things that are just toys where it's fun to interact with the thing, because I feel like that actually sets a very high bar, a game can be fun because it has a story or it has cool characters or there's a scoring system or something. But a toy, it has to be fun to play with just because it's fun to play with just from the interactions themselves.

DEVON: Totally. We recently had an episode with Jonathan Blow, the video game designer, and he's really obsessed with the performance of his video games and making sure they feel really good. He was making a similar point that, if it doesn't feel really good, people will just leave because they don't need to be playing your game compared to work software or something that their boss made them use. They have to use that.

OMAR: Right. I was looking at, what are the haptics like in Android phones? As far as I know, they're really not even close. Apple has spent a considerable extra money, and that maybe isn't justified on paper to have ... Because it takes extra space in the phone, I think, to have a really good haptic motor. That's obviously at a premium.

DEVON: It's not something that they talk about either. It's not in any of the advertising, anything like that. But you go into the Apple Store, you pick it up and you just feel it instantly. If we were to focus more on the history of computing on I/O devices, like terminals, displays, printers, all of those things, instead of on computers and sort of computing history that as it's normally done, how would that change the way that designers create interfaces?

OMAR: I think it would open up a lot of this discussion that we've had. I think a lot of the reason why there hasn't been much movement in computer interfaces ... There was the iPhone and there's the desktop computer. Basically, those interface paradigms have been the same since each of them was introduced. Your desktop right now is not really that different from the 1984 Mac. I kind of have this hypothesis that a lot of the reason for that stagnation is that there's been a stagnation in I/O devices. That you have your monitor and you have your keyboard and you have your mouse or your Trackpad. With the constraint that you have a monitor, keyboard and mouse, you can't actually do that much better than windows and icons and apps and stuff because the I/O devices actually impose quite a bit of constraint on what you can do.

DEVON: Why has there been a stagnation in I/O devices?

OMAR: I guess this is kind of an industry or business question. I think we've gotten really good at making displays, keyboards and mice. Once you have a market where there's millions of people buying computers, you also need to be compatible with that market. It's hard to do really new things because everybody wants to have their work processed and their spreadsheets, and those are pieces of software that you have to support. Some of the examples of things that I think we're constrained away from doing are really rich spatial interfaces, like the sort of zoomable, giant canvas UI, I think, are pretty awkward with a mouse and keyboard. I think visual programming is pretty awkward with a mouse and keyboard where you have blocks or boxes with wires between them. A lot of the kind of weird ideas for programming that don't just involve typing in a text editor, I think are one of the reasons that they haven't taken off more is that it's really annoying to use these with a mouse and keyboard because you have to drag one thing at a time.

DEVON: Touchscreens can be a little bit better for this, like an iPad?

OMAR: Yeah. I think so.

DEVON: But then those also feel kind of clumsy. It can feel fine for relatively big blocks, but you can't do that much detail. One type of input device you've used a lot is projectors. What are the sorts of things that projectors lend themselves to that normal monitors don't so much?

OMAR: The first thing that comes to mind is that projectors let you work on a much bigger scale. As long as your protector is far enough away, you can cover a lot more area. If there is something that feels more humane or something about that, where the scale is more in accord with your whole body yet and you're not just kind of crouched around this desk like you are with a laptop or even with a piece of paper or a book. There's the big thing. I mean, you can overlay objects in the world. I mean, this is a lot of what we were doing at Dynamicland. I think actually a lot of the potential of projectors is a little bit underplayed. In my mind, the stereotypical use of projectors is still, you have your PowerPoint on the wall of your office and you're clicking through it. I think you could do a lot more. It's a lot easier to have multiperson interactions.

DEVON: You've mentioned Dynamicland a few times. Some people listening probably aren't familiar with what that is. Can you describe what Dynamicland is?

OMAR: Dynamicland is a research project where I worked for a couple years, which was started by Bret Victor. Who's this sort of interface researcher. The goal of the project is to create sort of communal physical computer. Dynamicland itself lives in this building in downtown Oakland. It has sort of this whole floor. The idea is when you go into Dynamicland, you can leave your laptop at home. Instead, the whole building is basically a single computer. The way that works concretely is that there are all these tables around the space, Dynamicland space and above each table there are cameras and projectors. Rather than having a laptop and a mouse and keyboard and a screen, you have objects on these tables that you're moving around and that's what actuates the computer and the computer can project information on the tables. Rather than having virtual objects like files and windows and stuff on your screen, you have physical objects. Which are usually right now pieces of paper, and you can move around those physical objects and point them at each other.

The other key part of the Dynamicland system, and I was talking to you before about how I think a lot of people miss this even if they come in and see it, is that it's a full programming system. All of the pieces of paper, all the objects on the table and the space, every object is a little computer program and you can take a keyboard off the shelf and point it at one of those objects and reprogram it to do something else. The goal is that everybody who comes in should know or learn how to program, and it's sort of just part of the social aspect of the space, that this is also a programming space. That's something that's really novel about it. There have been physical computing systems in the past where there's a whole wall that's activated or there're projection map objects, but I think this further aspect of you can also reprogram the system inside the system without having your laptop or your desktop, I think that's something that's really new.

OMAR: That's a kind of philosophical thing that I carry over to a lot of other things that I'm doing now.

DEVON: If you had a full Dynamicland set up at home, what would you use it for?

OMAR: Ideally, you use it for whatever you use your laptop and phone for right now. You can use it for your normal work. That's really the goal of the system, is to be a computer in the full sense of the word. It's not supposed to be just an educational system or a kind of toy. It's supposed to be the actual system that you're working inside.

DEVON: What are the current limitations of Dynamicland to reach that goal at the moment?

OMAR: I mean, I think it depends on your priorities and who you ask. I think there are, and this is what the team has been working on for the last while, is there are a lot of technical constraints and there's a lot of engineering to do in terms of making more of it reprogrammable. Allowing more kinds of objects that aren't just pieces of paper. Those are really important because I think when people go to a website or they see demos of something, they really anchor onto whatever concrete examples they see and they kind of extrapolate from that. Like what is this whole system about? I think having more kinds of examples of what you can do with the system is pretty important, and that's a lot of the technical work they're doing. I mean, a lot of the things that I ran into where I felt limited were thinking about things like, what do I do on my laptop?

That's like, I talk to my friends, I use Twitter, I read books and PDFs and articles. A lot of those things weren't possible in the Dynamicland system for various reasons. I think those are concrete examples of things that you might want to do. Where maybe you don't have the internet connectivity or you don't have the high resolution rendering or the tracking of physical books. Or whatever you would need to be able to do those kinds of tasks. Instead, it was very much bottom up. Like, we're making these demonstrations of things you can do in the system, but not necessarily hoarding workflows from the existing computer.

DEVON: What are aspects of the system that seem really important about Dynamicland, but are actually sort of incidental and could have been done in a totally different way?

OMAR: The things that people fixate on when they first see the system are the least important things. For example, there are these colored dots that are used to track the pieces of paper. If you ever go and see a picture of Dynamicland or a video or something, that'll probably be the first thing you notice, is the color dots. They're really this icon or logo of the system. Those are completely incidental and you could make a system that to me would be recognizably a Dynamicland based system and not have colored dots. Conversely, you can make a system that had color dots that does not have any of the interesting properties of the Dynamicland system.

DEVON: Let's change gears a little bit. You mentioned when you were talking about Dynamicland, about how it's social. I remember you've talked to me a lot about how CS107 II, the class that you taught at Stanford titled, "Computer Systems from the Ground Up" and the office hours for that were a very social space. What did that look like and how did people interact in that space?

OMAR: This was this sort of second or third class you take in CS. You get taught C and systems programming and stuff like that. It was this experimental version of the class where I set up meetings, or you kind of knew that there was this gathering point where people would be anyway.

DEVON: If I were to go back to college, I would try to filter on the class size and ratio of staff and faculty. I think I really didn't appreciate that as a student until maybe my second or last year of school, where I started taking more of the small classes. Those classes were almost always so much better. I wish someone had told me that when I was 18.

OMAR: Whenever you have a personal relationships that are involved where people know who you are and you're trying to impress them or you're friends with them or whatever, I feel like that creates a much stronger relationship with the material and it's a lot more motivating than if you're just part of this machine and you're sitting in this lecture hall.

DEVON: Definitely. I mean, I think I found this is true also for learning languages. I've become a lot more motivated in the last few years to really get good at Spanish because my significant other is from a Spanish speaking country. The idea of being able to speak with his mom is really incredibly motivating. Versus in school, it was cool. I mean, I enjoyed learning Spanish, but it was not in the depths of my heart.

OMAR: I guess it's funny because this way of thinking really, it makes me a lot less interested in a lot of things about education or pedagogy than I think I would have been maybe 10 years ago. Because it feels to me like that's never the important part. The important part is always the motivation part and the social part of why you're learning the thing. One of the biggest things is, if it feels like you have to switch modes to use the thing ... The example that I've brought up in the past is, when I was TAing this class often people would have these really nasty bugs in their programs. They would always, always, always put off using the debugger to track down that bug. We would always kind of try to push them to use the debugger and they would really put it off until the last minute. They would insert print statements or they would make the thing blink or whatever. That's really telling. That tells you there's something wrong with this workflow or interaction of switching to the debugger.

I think people do dread it and I think they dread it because it's like, you have to go off into this different world to invoke the debugger. You have your normal ways of doing things where you write your program in your text editor and you compile it and you send it to the Raspberry Pi. But now, you have to stop all that and you have to go use this weird debugger thing that has completely different commands, you run it in a different way. Also, you're kind of admitting to yourself that you can't figure it out. I think there's also a sort of ego thing to that, but this sense that you have to stop what you're doing, that it's this cliff and you have to do something different, I think that generates a lot of dread. I think that's often associated with tools like debuggers or profilers or whatever. Where it's outside your normal interaction with the programming system.

DEVON: Is that something that is sort of inherent to the entire idea of a debugger as we think about it? Or is it something that you think, if we tried to design a debugger with that idea in mind, we could make just a way better debugger.

OMAR: I think you could do a lot better. I mean, I think the word debugger is a little bit unfortunate. I think someone, it might've been my friend Will on Twitter was talking about this. The word debugger almost suggests this whole workflow already. Maybe you wouldn't call it a bugger. I think the basic question is, how can we see what our programs are doing and be aware of that and see why they're going wrong? I think the debugger right now, that's the thing that happens to be able to answer that question and give you that information. But you can imagine other ways maybe that information is just sort of ambiently present when you're running the program and you just sort of see things pop up and you don't have to explicitly go in and interact.

DEVON: Also one really big step, it's just even thinking to use it. Because it's related to what you were saying about having to go into a different mode or a completely different tool. Where when you're in one tool, you're in flow and one step kind of naturally leads to the other. You see something on the screen and you react to it. To go into a debugger, you have to stop yourself and be like, "Hey, this new thing called a debugger would be useful right now. I should go use that." It's also just, it may not even cross your mind because you don't have a strong habit buildup or any visual stimuli that would push you towards that.

OMAR: Right. There's also a threshold of, you're kind of trading off in your head, I think, the whole time, like, "Can I fix this without having to go into the debugger? Because if I fix it without having to go into the bugger, all I need to do is just add this incremental print statement or there's kind of things I can do at the margin that maybe ..." You're trying to figure out, like, "Is that enough or do I have to go into the bugger?"

DEVON: One of the things I really liked about office hours in college was, everyone would understand different parts pretty well and you would sit around a big table and work on problems together and talk about them. You would each walk out sort of having picked up the expertise from somebody else. You've talked about this sort of idea of local experts, or I think someone responded to one of your tweets saying it reminds them of heist movies. How there's the tiny person who crawls in the spaces and there's the big strong guy or whatever. Can you talk a little bit about that concept and your experience working with different people with different superpowers?

OMAR: I think the tweet in which this came up was talking about literacy and maybe driving. People often make this comparison that the programming is like literacy. It's like being able to read and write. I think what comes with that comparison is it's kind of a problematic view of what it means, of what literacy meant historically. Where not many people could read or write, and it was sort of a superpower to read and write, and it made you a better than everyone else. Or gave you access to things that you didn't have access to before. I think this is sort of a problematic individualistic view of what it means to have a skill. Where you're the guy with literacy and that lets you do all kinds of things. When really, and I think this was pointed out by this art historian on Twitter, often there would be someone in the community who is like the person who knew how to read and write and that gave them certain responsibilities for the rest of the community.

It's the community as a whole, maybe that is the right unit of analysis. The village or the neighborhood and the town or whatever, maybe that's the right unit of analysis rather than the individual person.

DEVON: In the past, we've talked about how hanging out in a system for a really long time can help you understand things about that system that you don't get from really shallow engagement. What's something that you learned by spending a lot of time in a particular system?

OMAR: I guess there's two systems that come to mind, but the one that I'll talk about first is this text editor, Acme, that I used for actually an entire summer. Acme is the built-in text editor for the Plan 9 operating system. Which was this research operating system from Bell Labs. It was sort of meant to be the successor to Unix and it was built in the '80s. Unix has Vi or Nano, it has these built in text editors. The Plan 9 equivalent of those is Acme. Acme is really interesting because you're supposed to use the mouse in it as much as possible. I kind of like this like subversive aspect of it that, rather than being this hardcore computer thing of you stay on the keyboard all the time, it's, no, you stay on the mouse all the time. I mean, you can find demos of it online. It's a much more flexible interface than really, I think almost any graphical program I've ever used.

Whereas you're working, as you're editing your files in Acme as you're programming, whatever you're doing in the text editor, you kind of build up this workspace and you build up this palette of the operations you've been doing. You can middle click on anything in the program to execute it. At first, you're typing in a bunch of commands and then middle clicking them to execute them. But over time, you've built up this palette of all the commands you've already typed in and you just click on them again to execute them again. You're actually developing this interface for the specific thing that you're doing right now. It just feels much more flexible and powerful and tuned to the hour or to the day that you're working. Rather than, I think most text editors where you can make a keyboard shortcut, but you're not going to do that unless it's something you really, really are doing all the time.

DEVON: I see. It sort of creates and builds up around you as you use it.

OMAR: Yeah. At very small scales too, not just at the scale of weeks or months or whenever you get around to customizing your text editor. You're kind of always customizing it all the time for what you're currently doing.

DEVON: You're doing it implicitly by just using it as opposed to having to make a decision?

OMAR: Right. I would almost compare that to what I think is cool about Screennotate, which is a screenshot app I made. Which is that you don't have to go off and do something else. You kind of just do the obvious normal thing, and then that just pays dividends. It's an app for Mac and Windows. I'll just describe the Mac workflow. Basically on your Mac, there's a keyboard shortcut you can use to take a screenshot and you can post screenshots on Twitter or send it to someone. Everybody who has a Mac can do this. What Screenotate does, is you install it on your computer and it hooks into that keyboard shortcut that you use to take screenshots. You don't have to do anything differently than what you're already doing. Now, once you have Screenotate, every time you take a screenshot, Screenotate will save it and it will recognize the text inside the screenshot. It'll recognize the title of the file it's from or the URL that the screenshot's from.

OMAR: Whatever kind of information it can find about where you got that screenshot from, it'll save alongside the screenshot. After you've had it installed for a while, you end up with this whole archive of things you've sent people or things you've posted or things you've saved. Like I have 30,000 screenshots, I think, now and you can search for particular phrases or you can search for websites that things are from and they just pop up immediately. It becomes this very powerful, flexible ... It's almost like a note taking system in a way, but it's all oriented around taking screenshots. Which is this gesture that you're already doing. If you're like me, you're already doing it all the time anyway and this just gives you all this extra functionality for free.

DEVON: I really like looking back in my Screenotate folder every once in a while. It's probably once every two months I'll just sort of peruse it and I'll be like, "That's right, I was thinking about this, or I was talking about that." It also has a social aspect where for me, it skews towards things I'm sharing with other people. It also reminds me of, "That's right. I was talking to my friend, Omar, about this." That's so nice. I always smile a little bit as I look through it.

OMAR: Yeah. A screenshot has all this rich emotional context with it. It's funny, I kind of wonder also if it's the fact that it is an image and it's from this very concrete ... If I have a screenshot of me talking to you in 2015 or whatever, that'll have the original Facebook Messenger from 2015 font and layout and stuff. Or if I have a screenshot from The New York Times, it would have The New York Times font from whatever year it's from. It is this extremely concrete image, and that, I think carries a lot of history with it. In a way that if it were copied and pasted into my notes or something, it wouldn't. The websites are often surprisingly brittle and the archive on your computer is often the thing that survived. Actually, this would be an interesting thing to check. I feel like I have a lot of screenshots where the original website is now gone and maybe my screenshot is the only archive of it.

DEVON: Every once in a while, Google Photos will give me a little push notification and be like, "Hey, we found a bunch of screenshots. You want us to delete them?" The sort of implication is, "They're useless of course. Isn't it so nice of us to help you throw away your trash." There's a part of me that's like, "Hey, that's almost offensive. This stuff is not garbage. It's actually really important to me." It makes me think that maybe other people don't use screenshots in the same way, or maybe Google is just really out of touch. I don't know.

OMAR: I feel like if you spend a lot of time on your computer or your phone, your screenshots, they're just as important as your photo gallery of photos you took in real life. It's what you were doing — that's what your screenshots mean.

DEVON: One of the phrases that you throw around sometimes when talking about Screenotate and screenshots and also other types of technologies and workflows, is the idea of folk practices. Can you explain what that means and what some examples are?

OMAR: I don't know if I have a definition of it, but it means the kind of things people do on the computer, especially end-users, like non-programmers, things people do that are not ... You wouldn't think of that as the right way to do this thing, but somehow a lot of people end up doing that. For example, taking a screenshot of something to save it instead of saving the webpage or copying the text somewhere, there is something that feels a little bit wrong about it. Because it's like the screenshot doesn't have the text in it usually and it's kind of a lossy thing and it's just this flat image. Or sometimes when I'm sitting on the train, people will come in and they'll take a photo of the map on the train wall with their phone and then the map is now in their photo gallery. I kind of think that that's kind of a folk practice because in some sense, the right way to get the map on your phone is to download an app or go on the train system website and get the map PDF.

But I think there's something to learn from that because it seems like that's actually easier and better than trying to get the PDF for that. There's a reason that people do folk practice.

DEVON: It seems related to trust to me, or at least the answers you gave, examples that you gave. Because maybe I had the experience of pulling up a map on Google Maps and it's actually really good and it gives me all the information I need. But I don't quite trust that Google Maps won't crash at some really important time. I take a screenshot of it just in case I lose signal or something like that. Whereas I'm pretty sure if I had complete trust that Google Maps would never crash, I would probably not do that.

OMAR: I think it's also the sense that, I also talk about this with files, there's this sense that a screenshot on your phone, this is an object. It's in your photo gallery, you know how to work with it. It has this physics that you understand. It has these operations that you know how to do and it's going to be there, you know how to get to it, and you know it's not going to suddenly go away or get lost if you happen to search for a different address like Google Maps might. I think that's what engenders trust. I think there's way fewer things than there should be on your phone or on your computer that have that kind of objectness to them.

DEVON: It feels a little bit, with really dynamic interfaces, they have all of these pros where they can respond to your input and all the obvious, nice things. But they also kind of feel like you're renting space or something instead of owning it.

OMAR: Yeah. Google Maps, I think is a great example actually, because it can only really be in one place at a time and you're kind of, at least I often find myself like juggling, like, "Do I want to give up my state here so I can go look up something else? Is that worth it?" I guess it's also this app model where the app is one thing and it takes over your phone screen and it can only have one state at a time. If you could have multiple copies of Google Maps or whatever. I mean, this is more common on the desktop, but still, I guess you've got multiple tabs or whatever, it feels very much Google is in control. You can do those kinds of things with photos or photos give you those kinds of properties that you don't have with normal software.

DEVON: For this next part, I'm going to list a few tweets, and then I'm going to ask you a question about them. Number one, what paint sign on unfinished parts of the program UI? It can smudge up your mouse pointer or your finger if you're not careful. Tweet number two, spending a month walking from one end of the code base to the other. Tweet number three, the unclosable tab. Tweet number four, a pop-up window as a kind of genie. These could go on and on. I actually collected 20 of them and I think there were probably even more that I just didn't find.

OMAR: I kind of been meaning to go through and ... I don't even know. Some kind of collecting activity. I don't even know what the upshot of it would be...

DEVON: What do all of these have in common?

OMAR: I think that they're all about trying to take stuff in the computer and give them some of the richness and texture and embodiment of things in the real world and scale also. Where everything on the computer is kind of pristine and closed and perfect and you can't touch it and it doesn't decay, and it all kind of fits in this 11 inch rectangle. I think a lot of what I try to do on Twitter is to try and subvert that or poke fun out or whatever, and say like, "What if you could walk around your code base?" It's like, your code base is more complicated than a lot of buildings that you walk around in. Why shouldn't you be able to feel that at your body scale or why shouldn't you have wet paint on your application if it's not done in the same way that a wall would have wet paint on it? Because I mean, it has the same property of it's not finished yet, and why can't you feel that in a visceral way?

DEVON: Yeah. The material doesn't really tell you about its state or ... Yeah.

OMAR: I guess like wet paint, that's kind of accidental, but I still kind of want it. It's not necessarily something that is desirable about the real world, but at the same time, there is something nice about having all these side channels and little ways to indicate information without people having to explicitly put signs up.

DEVON: What inputs do you put into your brain to come up with stuff like this?

OMAR: I mean, I think a lot of it is from the experience of living and working with systems like Acme or Dynamicland, I spent two years with. Acme, I spent a summer with. I think that gives you a very deep feeling of how you could do things on the computer and ways of thinking about computing. I think that's a feeling that you only get from really being immersed in the thing. I don't think it's something that you get from reading about the system or writing about the system. I think it's something that can only come out of immersion. I feel like now when I use the computer, when I'm on Twitter or I'm programming something or whatever, there's a part of me that has this constantly low level frustration about a lot of different things. That, why can't I just do this? Why can't I have these multiple things?

OMAR: Why can't I trust that the computer is going to work? When I'm programming, why can't I just see what's going on? Why can't I get the data that I need from this other place and have it here? More like see it so that I can see if my program's going wrong. I feel like a lot of what I tweet comes out of, I'm doing something and I have this feeling that I want this to work in a different way.

DEVON: What should we do to work towards those goals?

OMAR: A lot of what I do is trying to provoke people and make these little ... Sometimes they're demos. Sometimes they're just these striking images. I think that that's probably a first step. Is, I'm relatively opposed to trying to articulate a philosophy or have a manifesto or something of what should be done. I'm much more drawn toward like, "Okay, let's have this set of really striking really concrete images of, this would be something that would be cool or that would feel good in a way that we want." Having that in the back of your head as an anchor for what computing could be like. I think that's kind of a first step. I think it takes a lot of work to make new user interfaces and new ways of computing, and there are a lot of things that are in your way. Maybe we just have to work through it, or maybe there can be better tools to help us program stuff. But I think that is often the reason that we don't see more.

I'm much more drawn toward like, "Okay, let's have this set of really striking really concrete images of, this would be something that would be cool or that would feel good in a way that we want."

DEVON: Why do you prefer the striking images approach as opposed to the manifesto approach?

OMAR: I have this implicit assumption that this is the way people think. Is that they anchor on to, if they hear an image or an idea and they can picture it in their head, that's something that they'll keep coming back to. Whereas a philosophy or a set of rules, it's satisfying to write, but I don't know if it's really as creatively useful as just having a bunch of images. Then maybe later, maybe in a few years, I can articulate a philosophy, but I think that the images are the important thing.

DEVON: I think something with sort of verbalized principles is they can sound very clear descriptions, like clear tests that you can apply. But then when it comes to actually building something, it's a lot less clear if you are meeting that principle or opposing that principle. Company values at tech companies often have this problem where you'll be in a meeting and you're trying to decide between A and B and then one person's like, "Well, A follows our company values more because of blah, blah, blah." The other person's like, "No, B is actually more ..." Neither of them is totally wrong or at least as the principles are written. Versus I think if you have a very concrete example in front of you, you can be like, "Well, this thing is more like that thing and they share these characteristics." It's more concrete. Now, items are very high dimensional, so that starts to fall apart at some point too.

OMAR: I think principles are flaws. They're like the ashes or the excreted product of this really rich thing in your head and you try to write it down as principles, but you're really not getting most of it. Or if other people get it, that's because they've managed to reconstruct the thing that's in your head. But it's mostly not actually in their written text. In your head, where you have this really rich aesthetic sense, it's not actually an iron law, or you would be willing to trade it off against other things because you have this holistic sense of what you think is good. But when you write it down, it's a really reduced, rigid set of principles.

DEVON: Universalizing too. Computers, I feel like don't have to be this way, but the way they've kind of worked out in the last few decades is that they're very universalizing.

OMAR: Right. It's like there's one piece of software in a domain that everybody uses, millions of people use this thing. I think for computer kind of part of the goal is to mitigate or push against that.

DEVON: There's all the goals of hitting web-scale or whatever. Which is, I think a valid goal for a lot of different things, but I think it ends up undercutting a lot of smaller, more personal projects that you might do. I think Robin Sloan has a really good description here. I think you might've been the one who shared this with me saying like, "Why don't we have more software projects that are like a homemade meal?"

OMAR: Right. Yeah, I've seen it.

DEVON: You just make it for one or two other people. Or something like that.

OMAR: Often with these web-scale things, it's like they're way better than what was out there along one dimension or a few dimensions. It actually, there's good reasons to adopt them. But then you kind of lose a lot of other things. Or you're just forced to adopt them by management.

DEVON: I think a lot of them are really fantastic. The thing that bums me out is they suck some of the air out of the room. Because you can make a lot of money and have a lot of success by making "web-scale business" or something like that, and so it ends up being that we don't pay as much attention to other things the computer can do.

OMAR: Yeah. I mean, this also reminds me of, we were talking about I/O versus computation. I think a related point is that a lot of what people use computers for is not computation, but just to talk to other people, to communicate with other people. Whether that's email or messaging or making shared documents. I think the question of what software is on the computer is also a question about how do people relate to each other? How do people want to communicate? Maybe we don't want to give all that up to just a few pieces of software. Maybe people should have their own software, that that's the way that they want to communicate with the particular society that they're in.

DEVON: What is some software that you would like to build that would be more like a homemade meal?

OMAR: I mean, I think to some extent, Screenotate is like this. I mean, it's definitely this shining example of, I made this software for myself and I use it for myself and that's still, I think the first test of whether I want to make some change to it. It's like, is this going to benefit the way that I use it? Another example is, I have this browser extension, TabFS. One of the reasons I built it was because I was frustrated with the rigidity and the lack of extensibility of my browser. One of the directions that I want to go with it is using it to build little workflows or little extensions or scripts around the things I do on the computer that are much lighter weight than building a whole browser extension. Which I'm not going to do because it's a lot of work. TabFS hopefully would make those kinds of things possible.

DEVON: Can you share a little bit about what TabFS is? What's the sort of thing that you could do with TabFS that you couldn't do or it would be harder to do with a more traditional extension?

OMAR: Yeah. TabFS, it is a browser extension. What it does is, it exposes all the tabs that are open in your browser as files on your file system. They're like synthetic files, so every tab you have open gets turned into a folder on your computer and then there were files on it so you can get the HTML content of the tab and you can get the title and you can poke things inside the tab, send scripts to run inside of it and stuff like that. You can kind of go back and forth between your file system and your browser. What that means is you can script your web browser just using whatever scripting languages you use on your normal computer. They can just be one line scripts or they can be little files or whatever. You can get a lot of the effect of writing a whole browser extension by just writing these little lightweights scripts. I actually use it for a few things right now.

I have a little indicator in my menu bar that tells me how many tabs I have open. That's just looking at how many folders are in the TabFS file system. That's kind of nice. I just click it every so often and it refreshes and tells me that I have 30 tabs open or whatever. Then I also have some scripts. I write these email updates for my GitHub Sponsors thing, and I run the scripts and it turns into a screenshot of that tweet and gets embedded in what I'm writing. That uses TabFS to go do the communication between my text editor and the stuff in my text editor and the web browser where I have a tweet open. That communication would actually be pretty hard to do if I had to write a whole browser extension just to do that. That's really only possible or it's only worth the effort because TabFS already provides this communication layer where I talk to the browser from the rest of my computer.

DEVON: Something I would love for you to do, if you did a Twitch stream or some sort of video of you talking through these little tools that you've built here and there and showing them and showing them how you use them. Because I think with a lot of these things, this is also true for your work at Dynamicland, where I often and find that it's hard to follow exactly what's going on until I see a concrete example.

OMAR: Video is often a good medium for stuff like this. When you're writing or talking about something, you don't know what the other person doesn't know or you often miss important details that would actually help to fix their understanding.

DEVON: This is, I think, one place where screenshots really shine again too. Where there's a whole context that you get. Not just the imagery and the style, but you get to see what the other person was seeing as opposed to a description of what the other person was seeing.

OMAR: Without the screenshot, I think you often try to explain the conceptual model first and that doesn't quite ... It can be better to just be very concrete and give you a screenshot and be like, "These are the things you click."

DEVON: I'm going to ask you just one closing question. Which is, what are three ideas that you've set aside that you would like to pursue in the future?

OMAR: The first, I only set aside a few months ago, it's this thing called web fork, which is sort of a web browser that I was working on. The idea is that you can fork tabs in a web browser instantly in a few milliseconds. If you're in a page, you can just make a copy of it and you can click something and open it in a new tab. You can click anything and open it in a new tab because new tabs can just be duplicated from the previous tabs. That's hard to explain, but that's something that I want to get back to working on. It has this longer term aspect of it, of being a browser that's under my control where I can monitor and kind of visualize what these pages are actually doing and what are the connections that they're establishing to other places? I don't know. There's a whole direction there or there are a bunch of different directions there that I think are really interesting having operations that you can do to any tab and having visualizations of a lot more of what's going on than your computer normally shows.

OMAR: The second thing is, we talked a little bit about this, but I was messing with haptic stuff in the beginning of this year with the iPhone. There's a really great demo from Apple, and you've played with it too, where you can bounce this ball off the sides of your phone by tilting it. It's weirdly impressive and it's just this Apple example code. I've not actually seen anything in a commercial app that's as good as that, and I feel like that shows the potential of these haptic systems. I was also playing with the PS5 controller, which I think has mostly been reverse engineered now. Thinking about new interfaces that use those. I mean, they were very constrained, so I think you really have to be careful to come up with something that feels good. They're these very non-intuitive rails of what feels good and then what really doesn't feel good with these haptic systems. Also, they're designed for certain contexts, like moving a character around in a video game.

But I feel like there's still a lot of things you could do which would give you a much more dynamic development workflow. So you could work, like you're writing Python or Ruby or something, really, really quickly instead of waiting minutes or hours to compile your thing. Then also having languages that were domain specific that could represent specific concepts. This is only possible now because a lot of these have been reverse engineered and the hardware vendors are no longer able to dictate how you program them. I think that's the last of the three. That also relates to my interest, we didn't talk a ton about this, but my interest in computers where I really understand everything that's going on. I think this is one ingredient in that of having pieces of hardware, where you program the hardware, and then you can use those to interface with other pieces of hardware without needing a driver. I don't know if that makes sense, but that's kind of my vision of why that's useful or interesting.

DEVON: Can you say a little bit more about that?

OMAR: Well, this was something we ran into at Dynamicland a little bit. Where you have this room with a bunch of projectors and cameras and a bunch of computers. One of the questions is, okay, how do you talk to a camera? You have a camera pointed at the table so it can see pages. How do you get the pictures from the camera into your software on your computer? The answer is, you have the cameras plugged into a USB port and then you have to know certain API's in Linux and you ask it like, "Can you give me the image from the camera?" It's actually very complicated. You need to get a lot of things right for that to work. You're relying on this huge stack of Linux's millions of lines of code and the drivers, a ton of code, and you don't have control of a lot of things that you might want to have control of. Like the timing and the compression and all this other stuff.

If you controlled your own hardware, you could run it at the speed that you needed, and you could have a talk in the protocols that you want to be able to communicate with pieces of hardware like that. Or at least that's kind of the dream. The dream would be that it would be a much simpler end to end system because you wouldn't need Linux and you wouldn't need these weird drivers and you wouldn't need to have exactly the right pieces of hardware because you could just have these pieces of clay sitting in the middle of your system that you transfigured to do the job that you need them to do.

DEVON: It sounds like I'm going to have to have you back on the show at some point once you spend a bunch of time in these systems and learn more about them. All that stuff sounds really fun.

OMAR: Yeah.

DEVON: Cool. Well, thank you so much, Omar. This is a really fun conversation.

OMAR: Yeah, this is a lot of fun.

DEVON: This is really a good excuse to dive deep into your work, which I've always found super interesting and it's always been fun to talk about. But it doesn't happen organically. Thank you. See you soon, I hope.

OMAR: Thanks.