Children deserve a first-rate education; and the public deserves first-rate reporting on it.<\/h4>\n

Please support our journalism.<\/p>\n

Donate Now!<\/span><\/button><\/a><\/div>\n

<\/a><\/p>\n

In a classroom assignment last month, Minneapolis student Teeth Logsdon-Wallace explained how the Ramshackle Glory song \u201cYour Heart is a Muscle the Size of Your Fist\u201d helped him cope after an attempt to die by suicide. In the assignment, which was flagged by the student surveillance company Gaggle, Logsdon-Wallace wrote that the song was \u201ca reminder to keep on loving, keep on fighting and hold on for your life.\u201d (Photo courtesy Teeth Logsdon-Wallace)<\/p>\n<\/div>\n

The classroom assignment was one of thousands of Minneapolis student communications that got flagged by Gaggle, a digital surveillance company that saw rapid growth after the pandemic forced schools into remote learning. In <\/span>an earlier investigation, The 74 analyzed nearly 1,300 public records<\/span><\/a> from Minneapolis Public Schools to expose how Gaggle subjects students to relentless digital surveillance 24 hours a day, seven days a week, raising significant privacy concerns for more than 5 million young people across the country who are monitored by the company\u2019s digital algorithm and human content moderators. <\/span><\/p>\n

But technology experts and families with first-hand experience with Gaggle\u2019s surveillance dragnet have raised a separate issue: The service is not only invasive, it may also be ineffective. <\/span><\/p>\n

<\/span><\/p>\n

Related<\/h4>\n

Exclusive Data: An Inside Look at the Spy Tech That Followed Kids Home for Remote Learning \u2014 and Now Won\u2019t Leave<\/p>\n

<\/a><\/div>\n

While the system flagged Logsdon-Wallace for referencing the word \u201csuicide,\u201d context was never part of the equation, he said. Two days later, in mid-September, a school counselor called his mom to let her know what officials had learned. The meaning of the classroom assignment \u2014 that his mental health had improved \u2014 was seemingly lost in the transaction between Gaggle and the school district. He felt betrayed. <\/span><\/p>\n

 <\/span>\u201cI was trying to be vulnerable with this teacher and be like, \u2018Hey, here\u2019s a thing that\u2019s important to me because you asked,\u201d Logsdon-Wallace said. \u201cNow, when I\u2019ve made it clear that I\u2019m a lot better, the school is contacting my counselor and is freaking out.\u201d<\/span><\/p>\n

Jeff Patterson, Gaggle\u2019s founder and CEO, said in a statement his company does not \u201cmake a judgement on that level of the context,\u201d and while some districts have requested to be notified about references to previous suicide attempts, it\u2019s ultimately up to administrators to \u201cdecide the proper response, if any.\u201d <\/span> <\/span><\/p>\n

\u2018A crisis on our hands\u2019<\/h3>\n

Minneapolis Public Schools first contracted with Gaggle in the spring of 2020 as the pandemic forced students nationwide into remote learning. Through AI and the content moderator team, Gaggle tracks students\u2019 online behavior everyday by analyzing materials on their school-issued Google and Microsoft accounts. The tool scans students\u2019 emails, chat messages and other documents, including class assignments and personal files, in search of keywords, images or videos that could indicate self-harm, violence or sexual behavior. The remote moderators evaluate flagged materials and notify school officials about content they find troubling. <\/span><\/p>\n

In Minneapolis, Gaggle flagged students for keywords related to pornography, suicide and violence, according to six months of incident reports obtained by The 74 through a public records request. The private company also captured their journal entries, fictional stories and classroom assignments. <\/span><\/p>\n

Gaggle executives maintain that the system saves lives, including those of <\/span>more than 1,400 youth<\/span><\/a> during the 2020-21 school year. Those figures have not been independently verified. Minneapolis school officials make similar assertions. Though the pandemic\u2019s effects on suicide rates remains fuzzy, <\/span>suicide has been a leading cause of death among teenagers<\/span><\/a> for years. Patterson, who has watched his business <\/span>grow by more than 20 percent<\/span><\/a> during COVID-19, said Gaggle could be part of the solution. Though not part of its contract with Minneapolis schools, the company recently launched a service that connects students flagged by the monitoring tool with teletherapists. <\/span><\/p>\n

\u201cBefore the pandemic, we had a crisis on our hands,\u201d he said. \u201cI believe there\u2019s a tsunami of youth suicide headed our way that we are not prepared for.\u201d <\/span><\/p>\n

Schools nationwide have increasingly relied on <\/span>technological tools that purport to keep kids safe,<\/span><\/a> yet there\u2019s <\/span>a dearth of independent research<\/span><\/a> to back up their claims.<\/span><\/p>\n

<\/a><\/p>\n

Minneapolis student Teeth Logsdon-Wallace poses with his dog Gilly. (Photo courtesy Alexis Logsdon)<\/p>\n<\/div>\n

Like many parents, Logsdon-Wallace\u2019s mother Alexis Logsdon didn\u2019t know Gaggle existed until she got the call from his school counselor. Luckily, the counselor recognized that Logsdon-Wallace was discussing events from the past and offered a measured response. His mother was still left baffled. <\/span><\/p>\n

\u201cThat was an example of somebody describing really good coping mechanisms, you know, \u2018I have music that is one of my soothing activities that helps me through a really hard mental health time,\u2019\u201d she said. \u201cBut that doesn\u2019t matter because, obviously, this software is not that smart \u2014 it\u2019s just like \u2018Woop, we saw the word.\u2019\u201d <\/span><\/p>\n

\u2018Random and capricious\u2019<\/h3>\n

Many <\/span>students have accepted digital surveillance as an inevitable reality<\/span><\/a> at school, according to a new survey by the Center for Democracy and Technology  in Washington, D.C. But some youth are fighting back, including Lucy Dockter, a 16-year-old junior from Westport, Connecticut. On multiple occasions over the last several years, Gaggle has flagged her communications \u2014 an experience she described as \u201creally scary.\u201d<\/span><\/p>\n

<\/span><\/p>\n

Related<\/h4>\n

New Research: Most Parents and Teachers Have Accepted Student Surveillance As a Safety Tool \u2014 But See the Potential For Serious Harm<\/p>\n

<\/a><\/div>\n

\u201cIf it works, it could be extremely beneficial. But if it\u2019s random, it\u2019s completely useless.\u201d
\u2014<\/span>Lucy Dockter, 16, Westport, Connecticut student mistakenly flagged by Gaggle<\/span><\/i><\/p>\n

On one occasion, Gaggle sent her an email notification of \u201cInappropriate Use\u201d while she was walking to her first high school biology midterm and her heart began to race as she worried what she had done wrong. Dockter is an editor of her high school\u2019s literary journal and, according to her, Gaggle had ultimately flagged profanity in students\u2019 fictional article submissions. <\/span><\/p>\n

\u201cThe link at the bottom of this email is for something that was identified as inappropriate,\u201d Gaggle warned in its email while pointing to one of the fictional articles. \u201cPlease refrain from storing or sharing inappropriate content in your files.\u201d <\/span><\/p>\n

<\/a><\/p>\n

Gaggle emailed a warning to Connecticut student Lucy Dockter for profanity in a literary journal article. (Photo courtesy Lucy Dockter)<\/p>\n<\/div>\n

But Gaggle doesn\u2019t catch everything. Even as she got flagged when students shared documents with her, the articles\u2019 authors weren\u2019t receiving similar alerts, she said. And neither did Gaggle\u2019s AI pick up when she wrote about the discrepancy in <\/span>a student newspaper article<\/span><\/a> where she included a four-letter swear word to make a point. In the article, which Dockter wrote with Google Docs, she argued that Gaggle\u2019s monitoring system is \u201crandom and capricious,\u201d and could be dangerous if school officials rely on its findings to protect students. <\/span><\/p>\n

Her experiences left the Connecticut teen questioning whether such tracking is even helpful. <\/span><\/p>\n

\u201cWith such a seemingly random service, that doesn\u2019t seem to \u2014 in the end \u2014 have an impact on improving student health or actually taking action to prevent suicide and threats\u201d she said in an interview. \u201cIf it works, it could be extremely beneficial. But if it\u2019s random, it\u2019s completely useless.\u201d<\/span><\/p>\n

<\/a><\/p>\n

Lucy Dockter<\/p>\n<\/div>\n

Some schools have asked Gaggle to email students about the use of profanity, but Patterson said the system has an error that he blamed on the tech giant Google, which at times \u201cdoes not properly indicate the author of a document and assigns a random collaborator.\u201d<\/span><\/p>\n

\u201cWe are hoping Google will improve this functionality so we can better protect students,\u201d Patterson said. <\/span><\/p>\n

Back in Minneapolis, attorney Cate Long said she became upset when she learned that Gaggle was monitoring her daughter on her personal laptop, which 10-year-old Emmeleia used for remote learning. She grew angrier when she learned the district didn\u2019t notify her that Gaggle had identified a threat. <\/span><\/p>\n

This spring, a classmate used Google Hangouts, the chat feature, to send Emmeleia a death threat, warning she\u2019d shoot her \u201cpuny little brain with my grandpa\u2019s rifle.\u201d<\/span><\/p>\n

<\/a><\/p>\n

Minneapolis mother Cate Long said a student used Google Hangouts to send a death threat to her 10-year-old daughter Emmeleia. Officials never informed her about whether Gaggle had flagged the threat. (Photo courtesy Cate Long)<\/p>\n<\/div>\n

When Long learned about the chat, she notified her daughter\u2019s teacher but was never informed about whether Gaggle had picked up on the disturbing message as well. Missing warning signs could be detrimental to both students and school leaders; districts <\/span>could be held liable<\/span><\/a> if they fail to act on credible threats.<\/span><\/p>\n

\u201cI didn\u2019t hear a word from Gaggle about it,\u201d she said. \u201cIf I hadn\u2019t brought it to the teacher\u2019s attention, I don\u2019t think that anything would have been done.\u201d <\/span><\/p>\n

The incident, which occurred in April, fell outside the six-month period for which The 74 obtained records. A Gaggle spokesperson said the company picked up on the threat and notified district officials an hour and a half later but it \u201cdoes not have any insight into the steps the district took to address this particular matter.\u201d <\/span><\/p>\n

Julie Schultz Brown, the Minneapolis district spokeswoman, said that officials \u201cwould never discuss with a community member any communication flagged by Gaggle.\u201d <\/span><\/p>\n

\u201cThat unrelated but concerned parent would not have been provided that information nor should she have been,\u201d she wrote in an email. \u201cThat is private.\u201d <\/span><\/b><\/p>\n

<\/a><\/p>\n

Cate Long poses with her 10-year-old daughter Emmeleia. (Photo courtesy Cate Long)<\/p>\n<\/div>\n

\u2018The big scary algorithm\u2019<\/h3>\n

When identifying potential trouble, Gaggle\u2019s algorithm relies on keyword matching that compares student communications against a dictionary of thousands of words the company believes could indicate potential issues. The company scans student emails before they\u2019re delivered to their intended recipients, said Patterson, the CEO. Files within Google Drive, including Docs and Sheets, are scanned as students write in them, he said. In one instance, the technology led to the arrest of a 35-year-old Michigan man who tried to send pornography to an 11-year-old girl in New York, <\/span>according to the company<\/span><\/a>. Gaggle prevented the file from ever reaching its intended recipient.  <\/span><\/p>\n

Though the company allows school districts to alter the keyword dictionary to reflect local contexts, less than 5 percent of districts customize the filter, Patterson said. <\/span><\/p>\n

That\u2019s where potential problems could begin, said Sara Jordan, an expert on artificial intelligence and senior researcher at the <\/span>Future of Privacy Forum<\/span><\/a> in Washington. For example, language that students use to express suicidal ideation could vary between Manhattan and rural Appalachia, she said.<\/span><\/p>\n

\u201cWe\u2019re using the big scary algorithm term here when I don\u2019t think it applies,\u201d This is not Netflix\u2019s recommendation engine. This is not Spotify.\u201d
\u2014<\/span>Sara Jordan, AI expert and senior researcher, Future of Privacy Forum<\/span><\/i><\/p>\n

<\/a><\/p>\n

Sara Jordan<\/p>\n<\/div>\n

On the other hand, she noted that false-positives are highly likely, especially when the system flags common swear words and fails to understand context. <\/span><\/p>\n

\u201cYou\u2019re going to get 25,000 emails saying that a student dropped an F-bomb in a chat,\u201d she said. \u201cWhat\u2019s the utility of that? That seems pretty low.\u201d <\/span><\/p>\n

She said that Gaggle\u2019s utility could be impaired because it doesn\u2019t adjust to students\u2019 behaviors over time, comparing it to Netflix, which recommends television shows based on users\u2019 ever-evolving viewing patterns. \u201cSomething that doesn\u2019t learn isn\u2019t going to be accurate,\u201d she said. For example, she said the program could be more useful if it learned to ignore the profane but harmless literary journal entries submitted to Dockter, the Connecticut student. Gaggle\u2019s marketing materials appear to overhype the tool\u2019s sophistication to schools, she said. <\/span><\/p>\n

\u201cWe\u2019re using the big scary algorithm term here when I don\u2019t think it applies,\u201d she said. \u201cThis is not Netflix\u2019s recommendation engine. This is not Spotify. This is not American Airlines serving you specific forms of flights based on your previous searches and your location.\u201d <\/span><\/p>\n

\u201cArtificial intelligence without human intelligence ain\u2019t that smart.\u201d
\u2014<\/span>Jeff Patterson, Gaggle founder and CEO<\/span><\/i><\/p>\n

Patterson said Gaggle\u2019s proprietary algorithm is updated regularly \u201cto adjust to student behaviors over time and improve accuracy and speed.\u201d The tool monitors \u201cthousands of keywords, including misspellings, slang words, evolving trends and terminologies, all informed by insights gleaned over two decades of doing this work.\u201d <\/span><\/p>\n

Ultimately, the algorithm to identify keywords is used to \u201cnarrow down the haystack as much as possible,\u201d Patterson said, and Gaggle content moderators review materials to gauge their risk levels. <\/span><\/p>\n

\u201cArtificial intelligence without human intelligence ain\u2019t that smart,\u201d he said. <\/span><\/p>\n

In Minneapolis, officials denied that Gaggle infringes on students\u2019 privacy and noted that the tool only operates within school-issued accounts. The <\/span>district\u2019s internet use policy<\/span><\/a> states that students should \u201cexpect only limited privacy,\u201d and that the misuse of school equipment could result in discipline and \u201ccivil or criminal liability.\u201d District leaders have also cited compliance with the Clinton-era <\/span>Children\u2019s Internet Protection Act,<\/span><\/a> which became law in 2000 and requires schools to monitor \u201cthe online activities of minors.\u201d <\/span><\/p>\n

Patterson suggested that teachers aren\u2019t paying close enough attention to keep students safe on their own and \u201csometimes they forget that they\u2019re mandated reporters.\u201d On the <\/span>Gaggle website<\/span><\/a>, Patterson says he launched the company in 1999 to provide teachers with \u201can easy way to watch over their gaggle of students.\u201d Legally, teachers are mandated to report suspected abuse and neglect, but Patterson broadens their sphere of responsibility and his company\u2019s role in meeting it. As technology becomes a key facet of American education, Patterson said that schools \u201chave a moral obligation to protect the kids on their digital playground.\u201d <\/span><\/p>\n

But Elizabeth Laird, the director of equity in civic technology at the Center for Democracy and Technology, argued the federal law was never intended to mandate student \u201ctracking\u201d through artificial intelligence. In fact, the statute includes a disclaimer stating it shouldn\u2019t be \u201cconstrued to require the tracking of internet use by any identifiable minor or adult user.\u201d In <\/span>a recent letter to federal lawmakers<\/span><\/a>, her group urged the government to clarify the Children\u2019s Internet Protection Act\u2019s requirements and distinguish monitoring from tracking individual student behaviors. <\/span><\/p>\n

Sen. Elizabeth Warren, a Democrat from Massachusetts, agrees. In recent <\/span>letters to Gaggle and other education technology companies<\/span><\/a>, Warren and other Democratic lawmakers said they\u2019re concerned the tools \u201cmay extend beyond\u201d the law\u2019s intent \u201cto surveil student activity or reinforce biases.\u201d Around-the-clock surveillance, they wrote, demonstrates \u201ca clear invasion of student privacy, particularly when students and families are unable to opt out.\u201d <\/span><\/p>\n

\u201cEscalations and mischaracterizations of crises may have long-lasting and harmful effects on students\u2019 mental health due to stigmatization and differential treatment following even a false report,\u201d the senators wrote. \u201cFlagging students as \u2018high-risk\u2019 may put them at risk of biased treatment from physicians and educators in the future. In other extreme cases, these tools can become analogous to predictive policing, which are notoriously biased against communities of color.\u201d<\/span><\/p>\n

<\/span><\/p>\n

Related<\/h4>\n

Democratic Lawmakers Demand Student Surveillance Companies Outline Business Practices, Warn the Security Tools May Compound \u2018Risk of Harm for Students\u2019<\/p>\n

<\/a><\/div>\n

A new kind of policing<\/h3>\n

Shortly after the school district piloted Gaggle for distance learning, education leaders were met with an awkward dilemma. Floyd\u2019s murder at the hands of a Minneapolis police officer prompted Minneapolis Public Schools to sever its ties with the police department for school-based officers and replace them with district security officers who lack the authority to make arrests. Gaggle flags district security when it identifies student communications the company believes could be harmful. <\/span><\/p>\n

Some critics have compared the surveillance tool to a new form of policing that, beyond broad efficacy concerns, could have a disparate impact on students of color, similar to traditional policing<\/span>.<\/b> Algorithms have long been found<\/span><\/a> to suffer biases. <\/span><\/p>\n

Matt Shaver, who taught at a Minneapolis elementary school during the pandemic but no longer works for the district, said he was concerned that <\/span>racial bias<\/span><\/a> could be baked into Gaggle\u2019s algorithm. Absent adequate context or nuance,  he worried the tool could lead to misunderstandings. <\/span><\/p>\n

Data obtained by The 74 offer a limited window into Gaggle\u2019s potential effects on different student populations. Though the district withheld many details in the nearly 1,300 incident reports, just over 100 identified the campuses where the involved students attended school. An analysis of those reports failed to identify racial discrepancies. Specifically, Gaggle was about as likely to issue incident reports in schools where children of color were the majority as it was at campuses where most children were white. It remains possible that students of color in predominantly white schools may have been disproportionately flagged by Gaggle or faced disproportionate punishment once identified. Broadly speaking, Black students are <\/span>far more likely to be suspended or arrested<\/span><\/a> at school than their white classmates, according to federal education data. <\/span><\/p>\n