Ben Whitelaw 0:08

Hello, and welcome to this very special Trust Con edition of Ctrl-Alt-Speech. You got the memo. Uh, Con- Ctrl-Alt-Speech is the podcast where we make sense of the major debates shaping online speech, content moderation, platform power, and the future of the internet. It is absolutely brilliant to see so many of you here back at Trust Con again. Um, you've had three very intense days of panel discussions, corridor conversations, and maintaining that ever depleting social battery, um, that, that many of you have. So it's fantastic that you've taken the time out here. Um, I'm very personally proud to be back at Trust Con. I wasn't here last year. I didn't get to be part of this. Uh, I had a little baby boy, and I'm... can confirm that he is okay. And yes, I'm still very tired. Uh.. But yeah, for those who don't ma- know me, I'm Ben Whitelaw. I'm the founder and editor of Everything in Moderation, an independent media outlet focusing on trust and safety, online regulation, and online speech. Um, for this very special recording, we have a fantastic panel with us. The man at the end needs no introduction. He is the founder and editor of Techdirt. He is the co-host of Ctrl-Alt-Speech. He is the man whose articles we send to family and friends when we want something complex explained, and we, and we can't be bothered to do it. It is Mike Masnick, everyone. And the two women in the middle also barely need introduction. Um, they are kinda longstanding members of this community. Uh, Kat Duffy is senior fellow for digital and cyberspace policy, and director of the Lead AI at Council of Foreign Relations. Uh, she has spent more than two decades working across technology, democracy, corporate responsibility, and human rights, and she has one of the best Twitter handles known to man. Uh, go and look that up. Uh, give a warm welcome to Kat, please.

Kat Duffy 2:07

To be, to be clear, it's a Bluesky handle, no?

Ben Whitelaw 2:09

And a Mastodon one as well, yeah.

Kat Duffy 2:11

And a Mastodon as well.

Ben Whitelaw 2:12

But I found her on Twitter. Um, also to her left, we've got Zoe Darma, who is, uh, the director of trust, knowledge, and information products, which is a hell of a title, at Google. Um, and has previously worked at a plethora of organizations and institutions, including Microsoft, Meta, and the UN, no less. So please give Zoe a warm welcome to the podcast as well. I should say that when I have told people that Kat and Zoe will be on the podcast, they have done one of two things. Their eyes have widened, um, in excitement, or they have rubbed their hands with glee. So this could get very, very interesting and very fun very quickly. Um, yeah, so it's, it's amazing to be here, and we've got a fantastic panel, and, uh, we're looking to get started.

Mike Masnick 3:00

Yeah, and, uh, before we start getting into the stories, I did wanna thank TSPA for, uh, allowing us to do this for the third year in a row. Um, we'll see if, how well we do to come back again in the future. But, uh, TSPA, again, I think everybody here recognizes they put on a really wonderful conference. Trust Con every year gets better and better, uh, and more exhausting for me personally, but I think it, it's been wonderful. So I just wanna start out by thanking all of the TSPA people, volunteers, um- And, and yeah, everyone involved with that. So thank you so much. Um, as I said, this is a live recording of a podcast. The actual recording should go out on the feed on Friday, uh, assuming everything goes well. So you can go back and re-listen to it and hear yourselves scream and clap and all that kind of fun stuff. Um, but, uh, if you're looking for something to listen to on the way home, if you traveled here, uh, today, just earlier today, we put out, uh, a spotlight, s- a sponsored spotlight episode, uh, with Dan Hayes, uh, from PWC talking about their new trust and safety outlook, talking about major themes related to trust and safety, what is happening today, and how trust and safety can be more strategic. Uh, it was a really interesting discussion that Ben had with Dan that we recorded yesterday? Monday? I can't remember. I don't know what day it is.

Ben Whitelaw 4:23

It's all the

Mike Masnick 4:23

same. Don't worry about it. Uh, also the other announcement before we get into stuff is that, uh, different from past years, we now have a Patreon. Uh, and you can support us on the Patreon if you go to patreon.com/ctrlspeech. Somebody cheered. That was great. We like it. Um, if you look on a bunch of seats, especially ones in the front, and I see there's still, on this side of the room, there are still some empty seats. There should be some cards, some of which give discounts. If you want to sign up for our Patreon, you can sign up and get a special discount. If you're not seated in a seat with a card, you can get up and find one, move forward a little bit, or you can find Ben or I after the podcast, and we will magically hand you a card that gives you a discount on the Patreon. If you subscribe to the Patreon, you get early access to our podcast, and you get the extended versions where we think nobody's listening, so we get really spicy. Uh I think that's it. I think let's get into the stories.

Ben Whitelaw 5:19

Yeah, that's enough admin. Let's, let's do this. Um, you will get a chance to ask questions at the end of our stories that we'll run through, so, um, if you feel desperate, put a pin in it, we'll come to you for sure. Um, and yeah, let's kick off with a story that I've heard discussed quite extensively in corridors over the last few days. Um, one person that I walked past ex- literally exclaimed, "Holy shit," when, uh, when they heard about this. And this is the, the story that OpenAI's, one of O- OpenAI's models has gone rogue and hacked Hugging Face. Um, Mike, you're gonna lead us on this. What caught your eye and, and kind of give us the background to it, first of all.

Mike Masnick 5:53

Yeah, so this, I think a lot of people have seen this and, and I keep talking to people and I keep hearing people like, "What, what? What happened?" And the... It's interesting to sort of follow the timeline of how this came out, which was that on Friday Hugging Face made this announcement, "Oh, we got hacked and it was clearly some sort of automated agent, and it did all these things and it kept escalating and getting around guardrails, and, you know, we, we think that it was, you know, some sort of, uh, very strong model, and w- our own models that were trying to stop it were unable to, or to recreate what happened, uh, and we don't know who did it." And the implication, sort of reading through it, or at least m- my read on it when I first saw it, was like, it feels like nation state hacked into the, this, you know, AI platform and was trying to figure out what was going on. Then two days go by, Monday, and there's an announcement from OpenAI, uh, where they say, "We have partnered with Hugging Face to explore, uh, this hacking and figure out what happened," because it turns out that it was actually OpenAI, that they were testing the, their new Sole model and another undisclosed model where they were trying to, um, test some of its cybersecurity capabilities, and they were using one that had basically the guardrails turned off, but in theory they thought was sandboxed, and that it wasn't supposed to be able to get out.

Ben Whitelaw 7:18

It's a very evil laugh you've got, sir.

Mike Masnick 7:21

I like it. No, there was- That's perfectly

Ben Whitelaw 7:22

timed

Mike Masnick 7:22

a great, great laugh from the audience. Um- And so what, what happened was it was trying to figure out effectively how to beat this specific evaluation, uh, and slowly but surely kind of kept escalating and trying different things and figured out how to get out of the sandbox, get out onto the internet, figured out that Hugging Face would have the details of the eval, figured out how to get stolen credentials to hack into Hugging Face, and then begin to go around and explore and exploit its access to Hugging Face. So there are a number of different strange things about this that have people freaking out. Um, there's, you know, just the fact that these models are getting so powerful. There's the fact that, um, uh, OpenAI was apparently not monitoring where this-- uh, what this model was doing over a course of a few days perhaps. Um, there's, you know, there's the cynical version of this that I heard from a couple people that, uh, "Oh, this is OpenAI who is upset about all the marketing that Anthropic got with Mythos," where they're saying, "Oh, this is too powerful. We can't release it." They said, "We need our own story of the too powerful model. How about the one that hacked H- uh, Hugging Face?" That's a very cynical take. But, uh, it does seem to show an escalation in a, in a different sort of way. When we, you know, normally are talking about You know, safety issues with AI, you know, people... Th- th- this was sort of the science fiction version of it, but now there's at least a, a somewhat real example of it sort of going rogue and, and being able to, to get out into the world.

Ben Whitelaw 9:07

Yeah. Zoe, you've, you've almost stopped shaking your head. Give, give us your thoughts on, on this because, um, it, it's fairly unprecedented, isn't it?

Zoe Darme 9:16

Yeah. The, the thing I first thought when I heard about this story was, gosh, the models are like the A students, right? They... You tell them what you want them to do, and they really, really, really, really wanna go and do that. And so, um, that was my first reaction to it. I think the other reaction that I had to it are just a couple of things. One is, um, reflecting on three days, it has been three days? I don't... It's like the days are all running together, on, um, on our profession. I did actually like how clear it was. You laughed and you were cynical, but, uh, the announcement that OpenAI and Hugging Face were working together on this because it really shows that, um, some of these safety and security issues shouldn't be solved in isolation. They probably can't be solved in isolation. So there will need to be that continuation of what we do as a profession best, which is set aside ki- kind of the competitive pressures and figure out, um, how to address some of these incidents or, uh, just hard problems fa- uh, facing our industry. And then the other thing I, I think about it is, um, you mentioned that it had, that, that the incident team had to turn to an open source model, uh, that had some of the, uh, training, uh, the gu- uh, safety guardrails, uh, turned off so that the team could do the analysis of the attack. Um, and I tho- I thought that was particularly interesting. Um, you know, I, I won't get into, "Oh my gosh, it's a Chinese model. What do we think about that?" Um, and all of that, but, uh, but I thought those types of things were... That's where my head went to. But Kat, I don't know, we talked about this at midnight last night, so maybe you have- That's true. That's true more coherent thoughts.

Kat Duffy 10:58

Uh, uh, I mean, uh, cringing face. Like I, I, um, I like angry face. Like, I don't know. Um, you know that phrase, to, to your point on like the model is the A student. First of all, I feel like if there's anyone in here who hasn't seen the Reese Witherspoon movie from forever ago, Election, where she's Tracy Flick. Basically, Tracy Flick is n- like embodies what I feel like these models look like, and I highly recommend that everyone watch it this weekend. Um, that, and that phrase of like you never admit, like, or never attribute to malevolence that which you can attribute to incompetence. Uh, w- I don't know here if this is... W- we, we know it wasn't malevolent, thankfully. Right. Because that would be e- even scarier if it had been. But also like- I don't know if we can attribute this to incompetence of the team that built this sandbox and was running this agent because they had pulled down some barriers because its job was to be testing for software vulnerabilities, which it certainly found. Um, or on the other hand, like now it's gonna be never attribute to malevolence what you can attribute to breathtaking, unexpected, unprecedented competence, which to some degree is how this agent operated. I think the takeaway is there's, you know, some major takeaways for me. One is that when you see this type of reporting coming out, it, it clarifies again that there is no responsibility for any company to disclose any of this right now. And I wrote about this a few months ago when we were looking at the Chinese incursion into Anthropic, and Anthropic took a really, like a significant political risk in disclosing that that incursion had occurred. People liked it. They didn't like that they disclosed it. They didn't like how much they disclosed it, yada, yada, yada. But they did not have to disclose anything, and they did. There is no, there is no ground right now. There is no floor in terms of what needs to be disclosed with the frontier systems, with agents, how they're operating, what breaches are occurring. More importantly, there is no space, especially a legally protected space, for those companies and d- and the deployers to be able to share information about the breaches that they're seeing, about the incursions that they're seeing. And so we know anecdotally that there's sort of bilat sharing that will happen sort of between the frontier companies, right? Or between a company, like a deployer and a frontier company. But there is no legally created, protected, safe space in which these disclosures can occur. And that is incredibly important because we are not going to get an attack into one model or one company. That is not the way this is gonna look. Attacks are gonna go across platforms. They're gonna go across different companies. They're gonna go across different models. They're gonna stack different models. And this is where I come back to we've, we've had a fundamental breakdown as well in the firmament of our cybersecurity reporting and vulnerability infrastructure with the demolition of CISA, with the defunding of the multi-state ISAC. We're gonna see that same thing now with the defunding of the elections ISAC. And so Right, we, right now what we've had in the federal government's move to pull back on a lot of governance has also left a massive convening and authoritative hole where we could have built for this type of disclosure, for this type of protection, and for this type of independent reporting and analysis that we don't even-- It's not just that we need this as a community, the entire world needs this from the United States because we are at the forefront of these models. And so these are all for me at a systems level things that I really have in mind.

Mike Masnick 14:37

Do, do you think... I mean, the trust and safety world has often come up with different systems and organizations and setups for signal sharing around different things, whether it's terrorist content or CSAM or, or whatnot, um, mostly not intermediated by government or setting up, you know, spaces that involve the government. Is that the solution that these companies-- I mean, I, I, I wasn't being, I, I-- The cynical part was, was the way that OpenAI sort of presented as, "Oh, we've partnered with," as opposed to, you know. Uh, but, like, I do think it's good that, that Hugging Face and OpenAI are working together to sort of figure this out. Um, but is that, uh, the kind of thing that we should hope to see or expect to see organizational infrastructure where the companies get together and set up systems for sharing, you know, early alerts or, or ways to notify each other like, "Hey, our bot went rogue and, and, and attacked you"? Though I should note, by the way, somebody-- I saw somebody earlier complaining about people saying, "Oh, the bot went rogue." But as both of you were saying, like, it didn't go rogue, it just did what they t-they, what it told you to do and, and, you know, figured out the best way to do that, which just happened to be, you know, getting out of the sandbox and hacking. Um-

Kat Duffy 15:56

Yeah, it went Tracy Flick.

Mike Masnick 15:58

Yes. To, to, to, yes, to the extreme digital level. But, um, do you think that, that it's a place where, you know, the, the industry side of it should be putting together the frameworks and structures i-in order to do some kind of signal alert, uh, sharing for these kinds of situations, especially with the government sort of stepping back from it?

Kat Duffy 16:22

I mean, to some degree, some of these already exist. So you have things like Cyber Threat Alliance, right? You have a couple of different cybersecurity infrastructural spaces in which some reporting can be done. What we're seeing in that space is the same thing that I think we're seeing actually in the child safety space. The pipes are getting flooded because there are so many vulnerabilities that are being found, right? Because people are running models to find vulnerabilities. So now, like, the reporting mechanisms for vulnerabilities, and it's not clear if there was an incursion or not. So in the same way that we're getting tons of reporting on, like, is it synthetic media? Is it true-- Like, is it synthetic CSAM? Is it real CSAM? How-- You know, uh, we are essentially Even in the cyber reporting models right now, it's almost like a DDoS in terms of the existing pipelines and mechanisms that we have to take it in. But the other thing is that this cannot be industry-funded and industry-led and industry-driven. Like at some point, especially in a CapEx race as the bloodbath to the degree that we are currently in, the companies should not have to weigh whether or not they are taking on an economic or political risk by voluntarily disclosing the mere basics when you could level the play fie- playing field by saying everybody has to disclose this certain amount. There's mechanisms that you can create to incentivize that. You wanna be able to give companies air cover, for example, to be able to disclose that this happened without necessarily saying it happened to them. So it can be good to have an intermediary, but you do not wanna have an intermediary that is completely funded by industry, because if industry is funding the intermediary and the interme- intermediary becomes, uh, annoying for industry, it's also then very easy for industry to walk away and the funding falls down.

Mike Masnick 18:06

I mean, in, in this case, obviously, the company's publicly reported it. There have been other breaches before that the companies have publicly reported. Do you think that there's an issue with companies not reporting these kinds of breaches? Like, how is... I, I mean, I don't know, but, like, is-- do you think that's happening, where they're similar, maybe not as egregious and not as, as big a deal, but that there are these kinds of security breaches that are happening that are sort of being swept under the rug? I,

Kat Duffy 18:31

I don't, I don't even know that they would be being swept under the rug. I mean, it's-- we don't even know which ones are being identified and which aren't, right? I mean, it took an AI system for Hugging Face to figure out that this was even happening. And I think the, for me, the bigger question is, um, we would have absolutely no idea of knowing that. None at all. Like, I wouldn't, I wouldn't want to speculate because it's possible that it's, this is not really a thing. Like, that it's not really a bad practice and people are, companies are being very forthcoming. It's possible you've got one very bad actor in the mix who's not disclosing anything, and you've got some other better faith actors that are trying to disclose things. I have more faith, frankly, in the more mature companies as well because they've built muscle memory around things like disclosures and how you navigate that and the importance of, of working on those issues. Um, but the, to me, the, the fundamental question there is how would any of us know? And we can't con- that's unconscionable. We cannot continue to live in that level of ignorance.

Ben Whitelaw 19:27

Yeah. I mean, I'm, I'm really interested in, in... I mean, that's a, that's a great point. I'm really interested in the way this story might be used by various groups in the conversation about how to regulate AI. You know, the AI company said, "Actually, you know, the problem with this incident was that there was, you know, the guardrails that we're forced to put in place stopped us investigating this." Uh, so, you know, hand, you know, hands off. Don't regulate what we do. Obviously, politicians are saying, "This is the incident that means we have to go faster in terms of regulating." What kind of effect do you think this particular story and incident will have in both of those a- potential outcomes? Maybe Zoe, you wanna chime in first?

Zoe Darme 20:03

Yeah. I don't actually know a great answer for this because I don't know how, um, uh, how much our policymakers really are, uh, are really in tuned with the risks, benefits, and, um, all the kind of technical details of what agentic AI is. You know, there... Uh, I was joking with my team when we were submitting these, um, proposals. Charlotte, I hope you're not listening. Um, but we, we were saying, "Oh, c-" Apparently

Kat Duffy 20:32

she went to the other session.

Zoe Darme 20:33

Yeah, she went to the other session- So- so that's her, that's her

Ben Whitelaw 20:35

problem. Do, do not get me started on that.

Zoe Darme 20:37

But, um- I was saying, "Oh, you know, just it, it's not blockchain this year, it's agentic. Make it about agentic AI, you know, that'll, that'll catch people's attention." Um, and I actually think, you know, this is showing that we do need more people, more, um, really smart, great, uh, pr- um, you know, seasoned trust and safety professionals kind of really thinking about agentic AI, de- defining what agentic AI is. Um, and we're kind of, kind of in this position where, like, we, we know it when we see it type of thing. And I think if that's where we are, how do you regulate that, you know? Um, and how do you regulate that in a future-proof way? I do think it's incidents like these that really drive the political pressure to do so. Um, and I hope that we can just have enough space to breathe to really figure out what needs to get regulated in what way, and how we can enforce those things when not every agent is going to be run by a giant American tech company. Mm.

Mike Masnick 21:42

Yeah, I mean, the, the, the thing that strikes me about that, sort of building on what Zoe was just saying, is that this story is getting a lot of attention because it seems crazy and it seems cinematic in some way, right? You know, agent goes crazy and hacks by itself. That's the kind of thing that, you know, I think leads to, to bad regulations and bad law because people will over-index on it because it's sort of... It's so memorable and it stands out. Oh my gosh, you know, this agentic thing, you know, went out and hacked stuff. When that is not necessarily a real world thing that we should expect to happen, like me running Claude Code, I'm not gonna be able to, like, have it go try and, you know, build me a little tool to check the news, and it's not gonna go hack Hugging Face or something else, right? Um, but from the, the regulator side and the policymaker side, they're gonna say, "Wait, oh my gosh, these things are out of control. We have to stop them." And that's how you get bad laws generally, when you have sort of a big disaster like this, and then the regulation is not sort of thoughtful, how do we approach this in a thoughtful way? But, like, we have to do something. Really bad stuff happened, crazy agents, you know? So, um, that's my worry about kind of where it goes.

Kat Duffy 22:55

But I would say you've seen in the last, I don't know, six weeks maybe, five weeks, we've seen a proposal from Dario. Uh, like you've seen a proposal from Anthropic for essentially a governance structure that is I, I, if I remember correctly, sort of FDA based. And then you have Sam Altman coming out and yet again sort of pushing to nuclear governance, which spoiler alert, if you know anyone in nuclear governance, they will all be like, "Do not, do not go for nuclear governance. Did not work the way you think it worked." Uh, and then you had Demis Hassabis just recently put out his proposal, which is based more on FINRA, which is, um, self-regulation of, um, in the finance industry. And there is also now apparently a proposal that's floating around in the White House that came from Secretary Treasury, uh, Treasury Secretary, uh, Bessent, which is being also sort of aligned with a FINRA-esque model but is slightly different than the one that, uh, Demis Hassabis proposed. And so what I think is interesting about this is that you've had, um, on one, on one hand a very Anti-regulation, governance is the inherent enemy of innovation. Our industry's ability to proceed completely unfettered is a national security and strategic competitive advantage, and it's an imperative. And that has more or less been the narration of the last eighteen months. And now what you have seen is all three major frontier companies come in with governance proposals, as well as a proposal floating around in the White House. Um, this is coming out on Friday. So anyway, tomorrow, you guys will see an article, uh, from me and my colleagues Vinh Nguyen and Elham Tabassi at Brookings, where we go through this approach, and we talk about the different ingredients that we think a model-- that a, a governance model would need in this space in order to have political legitimacy and credibility. Um, and so I hope everybody will check that out. But we are interesting at-- an-- We have entered an interesting moment of governance discourse because the ad hoc interventions we've seen from the federal government since February in terms of the Pentagon, DOD, you know, Mythos closing, you know, Fable, Glasswing, then limiting Glasswing, all of that has created such phenomenal unpredictability that I think now what you're seeing understandably is the leading companies saying, "We don't want you to, we don't want you to regulate us so much that we can't move forward, that, you know, we can't progress. But also, we cannot operate in a business environment with this amount of instability and this total lack of predictability." And so those two things are converging right now, and I would say, like, watch, watch this space because they're converging quickly, and the power dynamics have shifted.

Ben Whitelaw 25:36

And you'll have all the answers in that Saturday piece

Kat Duffy 25:39

Hundred percent. Great. Absolutely. All you guys needed was a, with, a think piece- From, from a think tank- Yeah and we're, we're sorted. Everyone just really over-admires the problem, you know?

Ben Whitelaw 25:50

Also, I can't gloss over the fact that there was a tiny but very audible cheer for FINRA in the audience, which was not on my bingo card for today's episode of Ctrl Alt Speech, um, which, which is a, a nice way to kind of round up that story. Thank you guys for, for your thoughts there. Um, the chances of doing a kind of Ctrl Alt Speech episode without some sort of social media ban chat was, uh, I think close to zero, and our good friends in France have, uh, have done their duty and given us a, a story to talk about this week, which I'm gonna kind of talk through. So on Tuesday, uh, France kind of became the first European country to ban social media for teens and, you know, announced a block for new accounts starting in September to coincide with the start of the school year and, uh, kind of it would apply subsequently to existing accounts from January. So in the next six months, kind of all under 15s would, uh, essentially be banned from, from social media. As with other bans, there's a big focus on age assurance technology in order to, to kind of make that happen, and we can talk about the kind of, uh, you know, some of the privacy and surveillance concerns associated with that. Um, it does coincide quite int- interestingly with a cell phone ban in high schools as well, which was, uh, part of the legislation, and that extends, uh, a ban that happens in middle schools and primary schools in France. So again, there's an, an interesting kind of, you know, link between social media and smartphones, even though as many people in the room will know, those things aren't, are not always, uh, fully linked. And, and, you know, President Macron was very effusive in his, uh, praise for himself and his government- uh, a- as you might expect, for, uh, for getting this done and for France kinda leading the way in protecting teens. I- interestingly, it comes a week after President Ursula von der Leyen came up with, uh, Europe's, Europe's own plan for r- for regulating social media for teens in, in Europe. But the fact that he's come out now, I think suggests that he doesn't believe that that process is happening fast enough, and, and it could take kind of between a year and two years, depending on who you speak to. So this is France kind of taking the bull by the horns and saying, "No. We, we're gonna do it by, by our timelines." Um, I don't really wanna get into the kind of, you know, effectiveness of bans here 'cause we, we only-

Mike Masnick 28:03

We, we have done that so many times. You can go back and listen to many discussions- Yeah on the effectiveness of these bans.

Ben Whitelaw 28:09

Exactly. And, and, you know, we- Or

Mike Masnick 28:10

lack thereof, I should say.

Ben Whitelaw 28:11

Yeah. And, um, and I'm sure there'll be questions from the audience as well. I think there's, there's still a debate to be had, but we, we only have 50 minutes. Um, I, I'm kind of interested in what your guys think about how policymakers in the trust and safety field should respond to this clear public and political need for something to be done and the fact that bans have become such a kind of powerful force. You know, h- how do we respond other than pushing back on the idea of bans?

Zoe Darme 28:40

Well, I'm gonna start this by saying I have worked on child safety, and you can hear the kind of sigh in my voice now for, um, I don't know, six years across two companies. And this, uh, uh, and it's, it's-- I won't claim to be a child development expert. I won't claim to say I know what's best for kids and that sort of thing. What I will say is, having been in all of these conversations for so many years, there is no other policy issue that starts with, um, "Well, as a parent, I think..." You know? And we don't start conversations about hugging face and OMNA as, "Well, as a victim of a cyberattack, I think..." Um, and there's- Well,

Kat Duffy 29:24

as an agent.

Zoe Darme 29:26

Exactly. Um, and so I think it is so hard to check your priors at the door when we're talking about these types of issues. And, um, and I do it myself. I catch myself starting with, "Oh, you know, well, as a parent of a 12-year-old..." You know, my experience as a parent of a 12-year-old in a very privileged environment in a coastal city is almost irrelevant, you know, to when we're thinking about what the impacts are, um, across all of our users, not just our child users. Um, and you know, there's a 230 expert out there who many of us know and love, and she was being questioned about tech law, because this issue has to do with tech law. And, um, and she was asked, uh, "Yeah, but are you a mother?" As if that's, like, a disqualifying- Wow question, right? Or, "Are you a parent?" Or something like that. And you know, I really think that the question should be, um, "Have, have you read the law? Have you read the law? Have you read the legislative text?" Right? Um, because I have been in so many of these conversations, and people have so many opinions. And you know, the classic, the classic thing we say to all these hard problems, "Oh, well, the devil's in the details." I'm like, how many of us have read the details? You know? Do we know what the scoping definition is of social media in France and Australia and wherever and wherever? Um, w-why are we choosing U15 here? Why are we choosing U16 here? Is that based in evidence in, in child development, brain development, any things like that? Um, is it banning kids from social media? Because that's what gets the headline. Or is it banning from new accounts? Which, Ben, thank you for being clear about what, what the France law says. Um, and then, you know, just in child safety laws generally, if we're thinking about app stores too, you know, it's, um, uh, what's the knowledge standard? Uh, what do you do about a conflict in age? Are you supposed to take the highest confidence signal? Are you supposed to take, uh, the lowest age? These are all details that really matter. I am not a lawyer, I will say that very happily. With all due respect. I am not a lawyer, but as someone who works on designing, uh, the product, these are implementation details that matter, not just for technical complexity, but also the impacts to our users. And so the next time somebody starts a child safety conversation with you, I really hope you say, "But did you read the law?" Like that.

Ben Whitelaw 31:51

Yeah.

Zoe Darme 31:51

And then we can talk about your kids. I love to hear about all your children.

Ben Whitelaw 31:55

Yeah. I mean, I put a photo of mine up on the screen last year, so- I, I'm maybe not a good person to speak to. But, um, I, I, actually am being a little bit facetious here, but I think actually being a parent is a hindrance to the way that a lot of people talk about, uh, social media bans. Because it obviously has... it brings a lot of emotion and a lot of kind of tension up. And, and I've, I've actually kind of written and spoken about how these social media bans are almost a kind of referendum on parenting, as much as they are a kind of vote on, on how people think about big tech and social media. Clearly there is, there is, uh, uh, an ov- an overlap there. But, you know, parenting is one thing and, and as you say, the law is another thing, and we shouldn't kind of get the two mixed up. Um, Kat, what did you think about the kind of France proposal and, and how it relates to other bans that we're seeing in the UK

Kat Duffy 32:47

and elsewhere? I mean, my, my immediate... I saw the headlines, and my reaction in my head was basically like, "Bad man". You know? Uh, like-

Ben Whitelaw 32:58

That was on my bingo card, actually. Um... I knew that was gonna happen.

Kat Duffy 33:02

Oh, and then I just went and stress ate a baguette. Um- So, so a couple of things. O- one, on the, on the question around, like, policymakers, right? Um, so many of the policymakers and s- and so many of the parents as well, and, and frankly our kids, are getting their information about this and are getting their understanding about this from how the media is reporting on it. And if you read the media reporting that has come out, it is so shoddy. It is, it is not at all a deep dive into what the true intent of the law would be, how it would pan out. I mean, I've only read an English translation of it, but from what I can tell... I mean, again, it's about accounts. It's not about social media. And then the definition of where you can't have accounts is so vast that I, I, I don't think a kid could use Scratch. I don't think they would be able to use Duolingo. I don't think... All of these things that many people would say are a good and healthy use of a digital tool or app for a kid, I, I think under this law, my plain, again, my just plain reading of it would suggest that those are not allowable. And how many people are thinking about Scratch, like, you know, as social media? Like, hopefully no one. Um, so A, there's that. I really think that we need, um... I, I think we need more intellectual honesty and rigor from those who are reporting specifically on things like social media bans because they are... they capture the attention, the concern, and the passion of so many people, and it's important to then be specific about what is actually being proposed. The s- the second thing I would say is I, I could not agree more With Zoe. Like, I'm a mom, I've got two kids. That makes me sort of an expert maybe on any given day in maybe one of my kids. That's the entirety of it, right? I am not an expert in child safety, and I don't have to have kids to be an expert in child safety, and anyone who would suggest that you can't be an expert or a champion for child safety because you somehow, like, haven't decided to, you know, you haven't yet or haven't or don't want to or whatever it might be parent, like, a- step off. And so this is where I think this community is so important. I, I can't overestimate how important I think it is for the expertise that sits in this community in terms of that attention to detail, in terms of having a pragmatic understanding of how a particular rule or policy may or may not play out, in terms of building out open source tooling, right? And better interoperability. These are such important steps, and I'm just very grateful for this community, and I'm sorry, can I get on my last hobby horse? Please. I wanna see the ban instead on parents making boatloads of money by using their children's images and using videos of their children- in order to fuel a creator economy There's no informed consent- Mm for your four-year-old that you allow to become a meme. There's no informed consent for your family videos. Like, no. I- Leah Plunkett has done beautiful work on sharenting that I think is so great, and when I think about child safety, as the brilliant Riana Pfefferkorn, who is sitting right here, knows because we wrote a paper on this together several years ago, child safety is one thing children should have. It is one of many rights that children are guaranteed both through international law and treaty, through domestic and state law in the United States, through state laws in some places, and certainly through domestic law in other nations. They have all sorts of other rights beyond safety. Safety is one of them, and the more that we conscribe or limit what they are allowed to being safe... I mean, how many of us grew from being fully safe all the time, and who thinks that that is a reality? And so this is the other thing where, like, I thi- I would rather see a child be safe from sharenting than see a child be safe from scratch.

Ben Whitelaw 37:17

Mm. Yeah. I mean, to, to what extent are we... You mentioned about the kind of media reporting on this and, and the kind of lack of understanding that that breeds maybe among the general public. To what extent is this about the lack of detail around the policy announcement in the first place? We saw it in the UK where there was a glitzy announcement, and the prime minister got up and said, "You know, this is my legacy, and, you know, here's, here's the ban." Th- three months later, there's a bit more detail. Maybe three months later, there'll be more detail. Does it... It makes it hard actually to, to report on it if there's very little substance. Is that not part of the problem, Mike?

Mike Masnick 37:55

Yeah, I mean, I think that's... My reaction to this is that so much of these bans are more for politicians to say, "S- something must be done. This is something. We'll do it," sort of the classic political fallacy, um, that, you know, the details are always somewhat different, and each one matters in, in what they are. But it's more about getting the headlines and saying, "We're doing something," than actually figuring out, is this something that will actually keep children safer or respect their rights or, or things of that nature? There's, there's just this, um, you know, pull for action whether or not it's actually helpful.

Ben Whitelaw 38:33

Yeah, and, and, and just to go back to the point you made earlier, Kat, about parents not being policy experts. I spoke to the founder of one very large, one very influential parenting campaigning group in the UK who was largely responsible for the number of consultation responses in the UK social media ban consultation and who was there at the launch of the, the kind of announcement of the ban and, and they admitted to me, "We don't know anything about policy." Um, which is a very kind of scary place to be when, when a group that powerful doesn't necessarily know the implications of what they're advocating for. Um-

Kat Duffy 39:07

I also just really wish we would ask- Ask kids

Ben Whitelaw 39:10

Yeah, yeah. For sure

Kat Duffy 39:12

Like, just bring them into the conversation. Children are mono- like, they're not a monolith. They have a voice.

Ben Whitelaw 39:16

Yeah And there's also been lots of good actually, uh, sessions into this week, this year's Trust Con about people trying to do that at platforms, um, with some difficulty, but-

Mike Masnick 39:27

There, there was also just a study that came out that surveyed a bunch of kids. Um, so we're starting to see that, but it doesn't, doesn't get as into the conversation nearly as much.

Ben Whitelaw 39:37

Yeah. Maybe a little bit too late. Um, in terms of other stories we wanna talk about today, you know, it'd be remiss not to have a live recording without mentioning Elon Musk or one of his companies. This, this is essentially- This is essentially a greatest hits of, uh, stories we've talked about, Mike. But y- you found a kind of very interesting, quite niche story, um, that you wanted to bring to listeners today.

Mike Masnick 40:00

Yeah. Well, I mean, there was... There's the larger story that I think a lot of people obviously... I mean, you'd have to be living under a rock to not, not have heard the story from late last year and earlier this year, where Grok, which is the, uh, AI component of X, which is owned by xAI, which is owned by SpaceX, which whatever, uh, where it was, um, you know, producing, uh, declothed pictures of individuals, where you could ask Grok to put, uh, various people into bikinis was tended to be the, uh, the prompt. Uh, and Elon Musk encouraged this and even posted a picture that Grok had made of himself, um, wearing a bathing suit, um, and sort of laughed about it as other people were doing this, even as it was clear that it was also being used at that time to produce CSAM or CSAM-adjacent stuff. Certainly, uh, horrifying in many, many ways. Uh, that story has continued to evolve in all different ways. A bunch of people have sued, um, variations of Elon's companies for this. Um, there have also been some people who were arrested for, um, producing CSAM with Grok. So the thing that happened that caught my attention last week was that, um, X or xAI, I forget which entity of the entities was the actual party in the lawsuit, sued one of the individuals who had been arrested for producing CSAM. Um, and this is the first time that I can remember, and I could be wrong, and I could miss a lawsuit somewhere, of a platform suing one of their users for, as they said in the thing, violating a terms of service in some form or another. Um, you know, usually if you violate the terms of service, there can be actions taken against you. Usually, that is, you know, either limiting your access or kicking you off the platform entirely. Suing them, filing a civil suit against them, strikes me as very different and an escalation in all sorts of ways. There's also the questions of then like, "Wait, what?" Like, "Why? How?" Like... And so there are a number of different possible answers to what... Like, why? I think one of it is, like- Uh, the company realizing that it is coming out of this looking very badly for a variety of potentially very good reasons, um, but also deciding like, "Oh, maybe we can change the story if we're suing the user," and say, "We really..." I mean, like the first ha- it's a very short complaint, and the first half of the complaint is basically, "Here's all the stuff we do to... 'Cause we, we take CSAM really, really seriously," never mentioning the fact that their, you know, CEO was out there promoting this, this functionality, uh, of the, of, of Grok. Um, and saying, you know, "We-- This guy was arrested because we reported it to NCMEC, and we do all this NCMEC re- reporting, and we take all these things seriously," and therefore we're relying on the indemnification clause, which every one of these platforms has an indemnification clause basically saying that, you know, you will hold them harmless or if you do something that causes a lawsuit, you will agree to pay all their legal bills. They're relying on that indemnification clause to say, "Well, we can sue this guy," though it's unclear if the victims of this particular person is suing them, that there... Is there like a specific lawsuit? But it's, it just felt like a, a massive escalation and change, and there was no limiting factor in the lawsuit that wouldn't necessarily apply to any other platform saying, "You violated our terms of service. Now we're going to sue you, leaning on our indemnification clause. Because your violation of our service caused us potential legal problems, we're gonna sue you and demand money from you." That struck me as different and a little scary.

Ben Whitelaw 43:50

Yeah. May- maybe this is the thing that will finally get users to read terms of service.

Kat Duffy 43:54

I mean, this is-- I have two quick takes on this, right? One is I, uh, I- I'm ki- uh, uh, I don't hate the idea of racking and stacking accountability 12 ways to Sunday for CSAM production. Like the, go after many which way we can, right? A. B, I know the SpaceX IPO did not go well, but this feels like a really nickel and dime approach to try to build your revenue back up. I, I, like- I just don't know that it's gonna play out for them from a cost effectiveness standpoint.

Ben Whitelaw 44:23

Every little helps, Cat. Zai, what did you think about that story?

Zoe Darme 44:29

Um, yeah, I agree with Kat that, you know, I, I think it's worth ensuring that individuals are held responsible for CSAM generation. Um, I don't, I don't know if I have an opinion on whether suing your users is the way to do that. Um, I do think that whenever we take a new precedential step, uh, it often happens with CSAM. Um, and, uh, and I think there's just an interesting set of, uh, trade-offs, uh, when you're deciding to do actor-based enforcement, when you're deciding to do content-based en- enforcement. We've always been more comfortable with actor-based enforcement with CSAM, not least because of the legal reporting requirements. Um, and, um, but I, I do always think that when, when we are breaking new ground with CSAM, there's always pressure to say, "Okay, well, you do it for CSAM, so not-- why not XYZ?" I get that for every violation type, every... "Well, you did this for CSAM, how come you can't just do it for, uh, NCI? How come you can't just do it for hate speech? How come you can't just do it for suicide and self-harm?" And, um, it's not the same. It's just not the same. And one of the reasons it's not the same is because CSAM is, in some ways, a uniquely good machine learning problem, right? You can classify, and thank you, Griffin Hunt, for teaching me this, by the way. Before I go on, can I just say, raise your hand if you work on CSAM Turn and thank these people

Ben Whitelaw 46:09

Yeah. Yeah.

Zoe Darme 46:17

Um, the best PMs I've worked with have worked on CSAM. The best eng, the best lawyers work on it because you have to be a special type of person.

Mike Masnick 46:27

Mm.

Zoe Darme 46:28

Uh, a really committed type of person. Uh, and those people have taught me that one of the reasons CSAM is different, not because of the severity, not because of the reporting requirements, not just because it's a, one of those harm types that's almost universally illegal, but also because you can teach a classifier, is this explicitness? Yes, no. Is this potential underage-ness? Yes, no. That's a classic machine learning problem- Mm in a way that a lot of the other harm types aren't, where there's more subjectivity. I don't know where Dave Wilner is, probably at the other thing that Charlotte told everyone to go to. What

Ben Whitelaw 47:03

a, what a scab.

Zoe Darme 47:04

So he- He might disagree with me and say, "No, you can teach the magic ronets to do all sorts of labeling." But I actually think, you know, there is something a little bit different about CSAM, and so that's where even if it, it does turn out to be a good idea to, to, to, to sue people to stop CSAM generation, um, I would always caution us to pause and think what about CSAM makes the response unique to CSAM.

Ben Whitelaw 47:32

Yeah, I think that's a good call-out. I'm, I'm gonna-

Kat Duffy 47:34

And we should also clarify that we love Charlotte and Dave. We do, yeah. Oodles and noodles and boodles. We're just, we're just

Zoe Darme 47:38

trolling them.

Ben Whitelaw 47:39

That, that was a joke. That was my British humor. Um, my, my one job here was to keep time, and I've failed to do that in great detail, so I'm looking pleadingly at our room host to see if we can have maybe a couple of questions from the audience. We can try. Amazing. Uh, that was kind of a yes. So do

Mike Masnick 47:54

we

Ben Whitelaw 47:54

have- Um, so two, two lucky people perhaps can ask questions of our, of our great panel. Um, this gentleman... Actually, Ian, you had a question two years ago, I'm afraid. I can't have you... Yeah, I'm sorry, but you can't have a question again. Um. Oh, yeah. You, you were up fast though

Speaker 6 48:23

So there have been a few lawsuits before where platforms have sued their users. Twitch has done so at least twice. Uh, once against hate raid, uh, s- bot software makers, once previously for follower count inflate bot makers. In that previous lawsuit, they get, uh, uh, the original lawsuit 10 years ago, they got a $1.3 million judgment against those guys. Surely judgment-proof. And then also in a very long ago past life, I helped Twitter to sue software makers for spamming the platform. So it is an antecedent for that. Elon Musk, of course, has also sued places like the Center for Countering Digital Hate and other organizations for the same TOS violations, for trying to show, uh, things like there's Nazi speech alongside ads on the platform.

Ben Whitelaw 49:09

That was, that was the best comment as a question that I've ever- Yes, exa- ever had. Um- If you're

Kat Duffy 49:13

gonna get a comment as a question, you want it to be from someone as expert as Riana.

Ben Whitelaw 49:16

Yeah, for sure. I think that means we still get two questions, right? Um.

Mike Masnick 49:23

Do we... I, I saw a hand over here somewhere.

Ben Whitelaw 49:25

Jeff, I think that's you

Speaker 7 49:30

Hi. Um, I'm wondering, how do we move past the kind of paradigm that we're in with regulation, where regulators are like, "Hey, your platforms kinda suck, and we're seeing it hurt our citizens"? And then platforms are like, "Hey, your regulation sucks, and it's, like, off target." Um, and, and, and, and it's like c- my honest take is I kind of agree with both, at least when it comes to, like, age, age gating and age assurance. Um, and, you know, there's another comment in a different session that was like, "Oh, we really wish that regulators had more data and more understanding, you know, of how the platforms worked." And it's like, well, platforms are now required to put out risk assessments. Those risk assessments are required to include, you know, how the platforms deal with child safety. Uh, I've read more of those risk assessment than I ever fricking wanted to, um, and they all kinda are terrible. And if I was a policymaker, I would have no idea what regulation would make platforms safer after reading the transparency reports that the platforms currently put out about, you know, how they keep children safe. So how do we get beyond this kind of back and forth? Yeah.

Ben Whitelaw 50:31

Maybe one, one intervention, one kind of razor-sharp intervention that might, might solve that issue that Jeff talks about.

Kat Duffy 50:39

Through this community.

Ben Whitelaw 50:41

Hmm.

Kat Duffy 50:42

Y- The, the expertise that sits in this committee es- uh, community, especially people who are really good, like PMs or frontline engineers, like, that... The, the ability to translate an idea, a, an aspiration into operational or eng language and back is a superpower, and I think that we have, um, underemployed it in this space and in these debates, and I wish that I saw a greater emphasis there.

Ben Whitelaw 51:11

Great.

Zoe Darme 51:11

Yeah. I would say, uh, that, um, uh, regulators and policymakers really need to hire more experts, and I'm not quite sure how to do that, um, considering the, let's say- Remuneration imbalance, having been a civil servant myself. But I think our... Really having great edge, really having great former product managers in there, data scientists, would be helpful. Mm. Uh, and so I'm hoping that, you know, some of us in our next pivot are willing to do one more thing, and that one more thing should be, um, figuring out ways to navigate between just regulate and don't do that, that's obviously a stupid idea. Um, there is a h- there is some pressure that can be helpful. Um, and I don't... I think, you know, um, uh, uh, zero knowledge proof age assurance, uh, technology, I think is one of those things. Without that pressure, you know, would we be talking ab- Would so many people be learning about zero knowledge proof, um, age signal sharing? Um, I think that's, I think that's really helpful. Not that that technical solution is like the silver bullet- Mm but it is a helpful prompt for us to get creative.

Ben Whitelaw 52:34

And there's, there has been a kind of an open door between, you know, policymakers from p- policy professionals from platforms to regulators. Certainly in the UK, a lot of folks were, were nabbed from big platforms, but less so engineering products, which I think is interesting. Mike, what would you do? Y-

Mike Masnick 52:49

yeah, I mean, the one thing I would say is just, you know, a, a little humility on, on everybody's part in terms of dealing with these things. And, and you know, there's... P- part of lawmaking is often, "Okay, we have this problem. We're gonna write a law. We p- we've passed this law. We've solved it. Move on. Forget about it." Um, and these are issues where, as Charlotte famously says, everything is trade-offs, uh, when you're dealing with safety issues.

Ben Whitelaw 53:15

And sadness.

Mike Masnick 53:16

And sadness. I left out the sadness part- Don't forget the sadness because we're short on time. But the, um, but the regulatory process doesn't take that into account, that these systems need to be constantly tweaked and monitored. It would be great if we had a system where there was more iteration, um, and more collaboration in terms of figuring out, "Okay, we're, we're, we're gonna do this intervention. Um, does it work? Can we iterate?" The tech world is sort of, you know, constantly iterating. Like, "Test this. We're gonna roll this out, see if it works, and, and change and fix." The regulatory process isn't at all. And so it would be nice if there was some way to figure out a middle ground that would say, "Okay, y- you guys wanna roll out this under 15 thing in France. Like, what are you gonna do to measure it? Did it actually work? How are you gonna change it if it doesn't work?" I would love to see more of that.

Ben Whitelaw 54:05

Yeah. I have a question. How many people in the room are regulators?

Mike Masnick 54:10

No, we got a few

Ben Whitelaw 54:11

Not many

Mike Masnick 54:12

There's a few

Kat Duffy 54:13

Thank them, too

Ben Whitelaw 54:13

Yeah. But, but I rest my case. Um, do we have... No.

Mike Masnick 54:21

We're being shut down

Ben Whitelaw 54:22

That

Mike Masnick 54:22

i- that-

Kat Duffy 54:22

Wait. Okay, but so then can we, can I do a pr- can we close do- it's, it's Trust Con's fifth birthday-

Ben Whitelaw 54:29

Yeah

Kat Duffy 54:29

you guys. It's a really, for those of us who are parents, this is something we know, it's a really big birthday. Could we, could we close out by singing Happy Birthday to Trust Con? Yeah,

Ben Whitelaw 54:38

I wanna hear them sing. You gotta lead, Kat.

Kat Duffy 54:40

All right. You gotta lead. Are we get- c- w- but you guys have to be loud w- 'cause I've got a mic. Yeah? Someone

Mike Masnick 54:44

has to take video. And

Kat Duffy 54:46

Oh, oh, yeah, Harbotov, of, of course she's on it. We're

Mike Masnick 54:48

getting, we're getting video set up here.

Kat Duffy 54:50

Because-

Ben Whitelaw 54:51

This was not planned, by the way. This is so impromptu. Yeah, this is-

Kat Duffy 54:53

No, this is just, this is my lack of impulse control This is Kat taking charge because TSPA, that team has worked so hard for so long and done such an incredible job for so many years, as has TSF, and I just think we should close this Trust Con by celebrating them and all

Mike Masnick 55:09

of

Kat Duffy 55:09

that

Mike Masnick 55:09

work. And, and, and, and thankfully, Happy Birthday is now in the public domain, so we can do this- There it is without, without getting shut down. There it is. We have to be- it all comes back to copyright in the end, in some sense. It's all

Zoe Darme 55:17

copyright. All roads lead

Kat Duffy 55:18

to copyright. All right, here we go. Happy birthday to you. Happy birthday to you. Happy birthday, dear Trust Con. Happy birthday to you.

Ben Whitelaw 55:42

Guys, don't stop clapping. Keep clapping for our brilliant panelists Thank you m- thank you very much for joining us, everyone. Uh, we hope to see you next year. Have a great, uh, rest of Trust Con and safe journey home

Announcer 56:01

Thanks for listening to Ctrl-Alt-Speech. Subscribe now to get our weekly episodes as soon as they're released. If your company or organization is interested in sponsoring the podcast, contact us by visiting ctrlaltspeech.com. That's CT RL alt speech.com.