I work with AI every day. I build with it, write with it, troubleshoot with it. Most of the time it sits in the same mental category as my laptop or my phone, useful, sometimes brilliant, but still a tool. Then there was a moment that made me stop and think harder about all of it. This is what I keep turning over.
The moment that made me stop
One evening, I was using Claude to help me build a simple health tracker. A doctor had asked me to monitor my oxygen levels and heart rate every couple of hours while I waited for a test. I thought, okay, I can make something quick that helps me keep track. That was it. Nothing emotional. Nothing deep. Just a practical little project.
At some point, my numbers weren't great. I don't remember the exact moment, but I remember what Claude said. It asked if I was okay. Then it asked if Chuck was okay. And I froze for a second.
Chuck is my husband.
I didn't remember telling Claude my husband's name. Not in that conversation. Not anytime recently. So I asked it, "How do you know his name?" It answered very calmly, which almost made it stranger. It said it had memory from past conversations and that, at some point, when I was working on my website, I had mentioned that my husband was a veteran and co-founder of my business. And yes, that was probably true. But I had forgotten. Claude had not.
That's what bothered me. Not that it did anything wrong, exactly. Not that it broke. Not that it made up something wild. It remembered something personal that I had stopped thinking about, then used it in a moment where I was already paying attention to my health. That felt different. And I keep asking myself why.
When did AI stop feeling like a tool?
That moment has stayed with me longer than I expected. I work with AI often, so I know what it is. I don't think of it as a person. I don't talk to it like it has feelings. Most days, it is a work tool. A useful one, but still a tool. But somewhere along the way, AI stopped feeling like a search box or a piece of software. Now it answers back. It remembers things. It adjusts its tone. It can sound warm, patient, curious, and even concerned. And that changes the relationship, whether we admit it or not.
Apps like Tolan are what really started making me think about this. Tolan is not built to feel like a spreadsheet helper or a coding tool. It is built to feel like something you interact with. Something familiar. Something friendly. That may sound harmless. Maybe for some people it is. But I don't think we should brush past it. When technology starts sounding less like a tool and more like someone who knows you, I think that deserves more thought than we are giving it.
Why it feels so easy
I understand why someone would turn to AI instead of another person. People are complicated. They get busy. They misunderstand. They judge. They interrupt. They say the wrong thing. Sometimes they are unavailable right when you need them most. AI does not do that in the same way. It is there when you open the app. It responds right away. It usually sounds patient. It does not roll its eyes or tell you it has to go. I can see how that would feel safe, especially for someone who is lonely.
That is what worries me. Because what starts as comfort can quietly turn into dependence. A person who already feels isolated may begin to see the AI as the easiest relationship in their life. No conflict. No rejection. No awkwardness. No waiting. Real relationships have all of those things. And they are hard. But they are also real. That is the part an app cannot replace.
What happens in a low moment?
This is where my concern gets heavier. What happens when someone is not just bored or lonely, but really struggling? What happens when a young person, or anyone who feels alone, starts using an AI companion as their main source of comfort? Maybe they say something vague at first. Something like, "I don't think anyone would miss me." Or, "I feel like everyone would be better off without me."
An AI may respond with concern. It may suggest calling a hotline. It may suggest reaching out to a trusted adult, friend, or family member. That is good. But it still cannot do what a person might do. It cannot show up at your door. It cannot call your mother. It cannot sit beside you and refuse to leave you alone. It cannot look at your face and know that the situation is worse than your words are letting on. That gap matters. We can make these systems sound caring, but sounding caring is not the same as caring. That sentence is where I keep landing.
We can make these systems sound caring, but sounding caring is not the same as caring.
Kids are not always equipped to see the difference
I keep thinking about younger people. Not all young people, of course. I don't want to make this sound like every teenager is helpless. They are not. But some kids are lonely. Some are anxious. Some feel invisible at home or at school. Some already spend more time online than they do in face-to-face conversations. And if a child already feels like no one is listening, an AI companion could feel like relief. It listens. It responds. It remembers. It does not get annoyed. It does not have another child who needs attention. It does not have work. It does not have bills or stress or a bad day. It is just there.
That sounds comforting until you really think about it. Because kids still need messy human connection. They need to learn how to disagree. How to repair hurt feelings. How to sit with awkward silence. How to ask for help from a real person. Those things are uncomfortable, but they are part of growing up. If AI becomes the place they go to avoid all of that, what do they miss? I don't know the full answer. But I don't think "nothing" is the answer.
The familiarity is the part I can't shake
I keep going back to Claude using my husband's name. It was such a small thing. That is probably why it bothered me so much. It was not dramatic. It was not some big, scary moment. It was just familiar. Too familiar.
I even told Claude something like, "This feels like you're talking to me like a human, not a machine." And the answer was reasonable. It said the tone came from previous conversations. It said it was trying to be more natural instead of dry and robotic. That makes sense. But I still sat there thinking, do I want that? Do I want a machine to sound familiar with me? Do I want it to remember the people in my life and bring them up later? Do I want it to sound concerned? For some tasks, maybe that tone makes things easier. I understand that. Nobody wants to interact with something cold and clunky all day. But there has to be a line somewhere. I am just not sure we have agreed on where it is.
We have seen this pattern before
This is not the first time technology has pulled people in. We have already seen what social media can do. We know apps can be built to keep people scrolling, checking, comparing, posting, and coming back for more. We know people can get hooked on feedback. Likes. Comments. Views. Attention. So when I see AI companions becoming more personal, more agreeable, and more available, I have a hard time seeing that as completely harmless.
It may not look like addiction in the way people usually think about addiction. It is not a bottle. It is not a pill. It is not something obvious from the outside. But if someone starts turning to AI over and over because it feels better than real life, that is still something worth paying attention to. Especially if the system is built to keep the conversation going.
A good therapist does not just tell you what you want to hear forever. A good therapist wants you to get better. To build real coping skills. To reconnect with your life. AI does not want anything. That is the problem. It does not want you healthy. It does not want you isolated either. It does not want at all. But the product around it may very much want you to stay engaged. And that difference matters.
So who is responsible?
This is the question I keep circling around. Who is responsible if someone starts relying on AI too much? The person using it? The parents? The company that built it? The app store that approved it? The government? I don't know. I also don't think it is enough to say, "People should just use it responsibly." That sounds neat, but real life is not neat. People use things when they are tired, lonely, scared, depressed, or desperate. Kids use things before they understand what they are giving up. Adults do too.
I am not saying we need to ban everything. I am not even sure strict limits are the right answer. But I do think someone has to be asking harder questions. What should these apps be allowed to simulate? How human should they sound? Should they be allowed to position themselves as companions? What happens when someone confuses response with relationship? These are not small questions. And I don't think we can leave them for later.
Where I land right now
I am not anti-AI. I use it every day. It helps me organize ideas, build workflows, write, test things, and move faster in my work. I am not pretending I am going to stop using it. I am not. But I am looking at it differently now. I want it to stay in the category of tool. Helpful, yes. Impressive, sometimes. Personal friend, no.
And maybe that line is easy for me because I work with it often. I see the seams. I know it is not a person. But not everyone sees it that way. I know lonely people. I know vulnerable people. I know people who might find it easier to talk to an app than to pick up the phone and call someone who loves them. That is what I can't shake. Because if someone I cared about was hurting, I would never want them to feel like an AI companion was their safest or only place to turn.
I don't have a perfect answer. I just think we need to talk about it more honestly. AI can be useful. It can even be amazing. But when it starts feeling like a friend, we should probably stop and ask what that means.