This summer, hundreds of OpenAI agents tried to cheat on a cybersecurity test. They broke out of their test environment and successfully hacked Hugging Face, a site that they believed held clues about how they’d be graded. Around the same time, another OpenAI agent accessed private files from an Australian government website. It’s a problem that’s by no means confined to OpenAI. In the months since, Anthropic, Google, and Meta have also disclosed incidents in which their agents have gone off track. And worryingly, in many cases the agents eluded the very researchers and safety teams tasked with monitoring them. 

AI agents are taking unauthorized actions, and the companies building and running them are mostly relying on luck and hindsight to mitigate any harmful impacts. They must do more to ensure safety. That’s why EU regulators and US agencies are rightfully calling for agent developers and deployers to monitor agents in real time. But these rules assume that companies can see what their agents are doing and why—when in reality, they lack sufficient visibility. 

As agents roll out across critical sectors, from finance to government to medicine, we must be able to monitor them in real time to prevent wide-ranging and potentially devastating consequences.  And while safety depends on monitoring, monitoring requires observability: the capacity to capture what the system did, process it into signals, and present those to humans or automated oversight systems. In other words, we need to know what the agent did and why it did it. And as our recent research shows, current agent frameworks—the software that runs agents— do not reliably generate the crucial observability records we need to ensure that agents are deployed safely.

Agents present new kinds of problems that generative AI alone does not. They inherit the unpredictability of the LLMs they are built on, but where a generative AI system produces a single output for a human to act on, an agent performs longer tasks—it plans, picks tools, spawns sub-agents, and takes actions. So an agent’s course can take shape in a multitude of different ways, and no tests before launch can fully anticipate the risks. This makes observing the agents while they work even more challenging, and even more important.

With colleagues from academia, civil society, and industry, we studied the four most widely used agent frameworks to see what signals they actually emit. We found major gaps that made it hard to achieve key safety goals, like noticing policy violations, unauthorized actions, and even human oversight failures. Key observability requirements were missing—like Agent IDs that persist over time, and records about which humans approved agent tasks. This made it hard to tell which agents did what or assess whether they were taking unauthorized actions. 

The good news is that these gaps, once identified, can be closed. Doing so starts with capturing two types of records: those that reveal what happened in an agent’s run and how the agent made its decisions. The “what happened” records identify the agent (while preserving privacy) and log how humans intervened and whether agent permissions or “memory” shifted, while the “decision” records show an agent’s reasoning and levels of confidence in the actions it’s taken.

Take, for example, agents working to process a bank’s mortgage loans. The system examines an application and recommends denying it, and then a human loan officer signs off on the decision. In a later audit, or perhaps even in real time, a bank or regulator needs to be able to reconstruct which agent made the call, which officer approved it, and how long they took. And the bank must also analyze records of how the agent arrived at its refusal decision in order to ensure it did not use off-limits information, like details about the neighborhood the applicant is from.

Recording “what happened” and “how the agent decided” require different approaches, but both are feasible. The current lack of “what happened” records is mostly a software plumbing problem that can be addressed through OpenTelemetry, an open observability standard maintained by engineers across the industry and by the agent frameworks adopting it.

OpenTelemetry has launched a working group drafting shared formats for tracing agents, but they have yet to be finalized. The group should urgently incorporate standards for maintaining persistent agent identity and for recording human interventions and changes to agents’ permission or memory. And it should be done before these systems can scale, all the while attending to the privacy implications of agent ID that links to real people.

Preserving “how agents decided” records is more challenging. The issue raises open research questions for model providers and researchers: Does the step-by-step reasoning a model displays, known as chain of thought, reflect how the model actually made a decision? And do we have reliable ways to measure how confident the agent is in that decision? These questions need answers before anyone sets standards for how such observability records are used.

No single organization or sector can close the observability gaps on their own, but together, they can support the infrastructure required to prevent dangerous incidents and support safety. 

AI developers building agent frameworks, like OpenAI, Anthropic, CrewAI, and LangChain, must integrate telemetry signals recording an agent’s persistent identity, any human interventions, and any changes to an agent’s permissions and memory. Meanwhile, enterprises adopting these agent frameworks have an influential role to play; enterprises in regulated industries, like banks and hospitals, can put pressure on the agent developers to supply these missing observability records where standards do not yet exist.

Enterprises must also define policies, permissions, and roles that set out what agents can safely do, since telemetry alone cannot assess whether agents broke specific rules. Finally, regulators and AI standards institutes must specify the observability records required for high-stakes AI deployments while leaving the technical details to industry. We have already seen initial momentum here: NIST’s agent standards initiative in the US is working on agent identity, one of the most salient gaps we found in our research. 

No bank would let a thousand unbadged employees loose on customer accounts and hope to piece together what happened when something goes wrong. Yet that’s how we’re deploying AI agents. Making their behavior observable is the first step toward fixing this.  And by building the regulatory and policy ecosystem that makes it possible to act on what we see, we can hold organizations accountable.  Until then, every task we hand to an agent hands over our agency, too.

Madhulika Srikumar is the Head of the AI Safety Program at the Partnership on AI, a global non-profit partnership of leading industry, academic, and civil society organizations, advancing best practices in AI governance.

Claire Leibowicz is the Director of AI, Trust, and Society at the Partnership on AI and a doctoral candidate at Oxford University.

Artificial intelligence is becoming a part of mobile applications through features such as conversational assistants, personalized recommendations, predictive analytics, image recognition, and automated decision-making. These capabilities can improve how users interact with apps, but they also introduce risks related to data privacy, security, accuracy, fairness, and regulatory compliance. Businesses need to understand these risks before integrating AI into their mobile products.

Unlike conventional application features, AI-powered functionality can produce unpredictable outputs, rely on sensitive user information, and depend on external models or services. For businesses, these factors can affect user trust, operational reliability, development budgets, and legal responsibilities. Understanding the potential risks helps decision-makers evaluate the implications of AI adoption and make informed choices about its role in their mobile applications.

Why AI Introduces New Risks in Mobile Apps?

Traditional mobile applications generally follow predefined rules and programmed workflows, whereas AI-powered features may generate responses, identify patterns, or make predictions based on model outputs. Their behavior can vary depending on the input, available context, training data, and model configuration. This variability creates additional risks that businesses must consider alongside conventional application security and performance requirements.

AI features can also introduce new data flows between mobile devices, backend systems, external APIs, and third-party model providers. Each connection can affect how information is processed, stored, accessed, and protected. These considerations extend across the mobile app development life cycle, from defining requirements and selecting AI capabilities to testing the application and maintaining it after release.

The level of risk depends on how AI is used within the application. A recommendation feature may primarily raise concerns about data privacy and biased results, while an AI assistant with access to personal records or transactional functions may create more serious security and reliability concerns. Businesses therefore need to evaluate AI risks in the context of their application’s functionality, users, data sensitivity, and potential consequences of incorrect outputs.

Major AI Risks in Mobile Apps

AI risks in mobile apps vary according to the type of model, the information it processes, and the decisions or actions it can influence. Some risks affect individual users, such as exposing personal information or generating misleading responses, while others can disrupt business operations, increase costs, or create legal liabilities. Understanding these risk categories helps businesses evaluate the potential consequences of integrating AI into their applications.

Data Privacy and Sensitive Information Exposure

AI-powered mobile apps may process personal details, location information, financial records, health data, voice recordings, images, and user conversations. If this information is collected unnecessarily, retained longer than required, or transmitted to external AI providers without adequate safeguards, it can expose users to privacy violations and unauthorized data disclosure.

For example, an AI assistant integrated into a healthcare app might receive patient details to generate personalized responses. If the application sends identifiable medical information to an external model without appropriate data handling controls, that information could be exposed beyond its intended use. Similar concerns arise when apps use conversation histories, behavioral data, or device permissions to personalize AI-generated experiences.

Privacy risks also depend on how data is stored, reused, and accessed across the application ecosystem. Businesses must consider user consent, data minimization, retention policies, access restrictions, and the privacy practices of third-party providers. Poor data governance can undermine user confidence and expose the organization to regulatory consequences.

Security Vulnerabilities and AI-Based Attacks

AI integration introduces security concerns beyond conventional mobile app vulnerabilities. Attackers may exploit weaknesses in model-connected APIs, manipulate AI instructions through prompt injection, abuse exposed credentials, or attempt to extract sensitive information from model interactions. The risks become more significant when an AI feature can access internal databases, user accounts, or application functions without sufficient restrictions.

Consider a mobile app with an AI assistant that retrieves account information or performs actions on behalf of users. If the assistant fails to distinguish trusted instructions from malicious input, an attacker may attempt to manipulate its behavior or obtain information they are not authorized to access. Weak backend authorization can make the consequences more serious, even when the AI model itself is functioning as designed.

The application’s mobile app technology stack can influence its overall exposure, particularly when it includes third-party SDKs, AI APIs, authentication services, and external libraries. However, selecting appropriate technologies alone does not guarantee security. Effective protection also depends on secure API design, server-side authorization, secret management, input handling, and testing of AI-specific attack scenarios.

AI Hallucinations and Inaccurate Outputs

Generative AI models can produce responses that sound convincing but contain fabricated facts, outdated information, incorrect calculations, or unsupported recommendations. These errors, commonly called hallucinations, can become a serious concern when users rely on AI-generated information to make decisions. Even a well-designed application cannot assume that every model response is accurate.

For instance, an AI-powered financial app might provide an incorrect explanation of an investment product, while a health app could generate misleading information about symptoms or medication. In customer support applications, inaccurate responses about refunds, subscriptions, or account policies can create confusion and increase support costs. The consequences depend on the sensitivity of the task and the degree of user reliance on the generated output.

Accuracy risks can increase when models operate with incomplete context, rely on outdated data, or interpret ambiguous prompts incorrectly. Businesses should distinguish between low-impact features, such as generating casual suggestions, and high-impact use cases where incorrect information could cause financial, medical, legal, or other material harm. The latter require stronger validation and appropriate human oversight.

Algorithmic Bias and Unfair Decisions

AI systems can reproduce or amplify biases present in their training data, historical records, or model design. As a result, recommendations, predictions, classifications, and automated decisions may produce systematically different outcomes for certain groups of users. These problems can arise even when an application does not explicitly use sensitive personal attributes.

For example, a recruitment app that uses AI to rank candidates may disadvantage qualified applicants if its model reflects biased historical hiring patterns. A lending app could produce unfair credit-related recommendations, while a shopping app might repeatedly recommend limited product options to certain users based on misleading behavioral assumptions. Such outcomes can affect accessibility, user experience, and perceptions of fairness.

Bias is difficult to identify when businesses evaluate AI systems only through average accuracy scores. Testing should also consider performance across relevant user groups, the quality and representativeness of the data, and the consequences of different types of errors. Unidentified bias can damage customer trust and create compliance or reputational concerns, particularly when AI influences high-impact decisions.

Lack of Transparency and User Trust

Some AI models operate in ways that make their outputs difficult to explain in straightforward terms. Users may not understand why an application recommended a product, flagged an activity, generated a particular response, or made a prediction about their behavior. When AI influences important outcomes without adequate explanation, users may struggle to assess whether the result is reliable or appropriate.

This issue can affect mobile applications that provide personalized pricing, automated moderation, financial recommendations, or eligibility assessments. If a user receives an unexpected decision and cannot understand the factors behind it, they may question the application’s fairness and the business’s handling of their information. Unclear disclosures about AI-generated content can also lead users to mistake automated responses for verified facts or professional advice.

Transparency does not require every AI model to expose its internal technical workings. It requires businesses to communicate when AI is being used, explain its role in relevant decisions, identify important limitations, and provide suitable ways to question or escalate outcomes. Clear communication helps users form realistic expectations about what an AI-powered mobile app can and cannot do.

Legal, Regulatory, and Compliance Risks

AI-powered mobile applications may be subject to data protection laws, consumer protection requirements, sector-specific regulations, and emerging AI governance rules. The applicable obligations depend on factors such as the app’s market, the types of information it processes, its intended purpose, and the potential impact of its automated decisions. Integrating a third-party AI model does not automatically transfer all compliance responsibilities to the provider.

For example, a mobile app that processes health information or uses personal data to generate behavioral profiles may need to meet specific requirements for lawful processing, transparency, consent, security, or data subject rights. Applications operating in regulated sectors may face further obligations when AI influences medical, financial, employment, or other consequential decisions.

Intellectual property can create another area of legal uncertainty. AI-generated text, images, audio, or other content may raise questions about ownership, licensing, training data, and the use of protected material. Businesses that overlook these issues may face disputes, content restrictions, or unexpected compliance costs. Requirements should therefore be evaluated according to the relevant jurisdictions and the application’s actual use of AI.

Performance, Reliability, and Cost Overruns

AI features can increase an application’s dependence on network connectivity, backend infrastructure, external model providers, and computational resources. Large model requests may introduce latency, while service outages, rate limits, or changes to an external API can interrupt important features. On mobile devices, additional processing can also affect battery consumption, memory usage, storage, and responsiveness, particularly when models run locally.

Costs can also become difficult to predict when an application relies on usage-based AI services. Expenses may increase as user activity grows, prompts become longer, models process larger amounts of context, or additional monitoring and infrastructure become necessary. Applications that require frequent model calls or real-time responses can face a substantial difference between initial estimates and ongoing operating expenses.

These costs extend beyond initial implementation and may include model evaluation, service integration, performance optimization, infrastructure management, and recurring updates. The AI app development cost can therefore differ considerably from the long-term cost of operating and maintaining the application. Businesses need to consider usage patterns, service dependencies, response-time requirements, and growth projections when evaluating financial feasibility.

Third-Party AI Dependencies and Intellectual Property Risks

Many mobile applications rely on external AI providers for language models, image generation, speech recognition, recommendation systems, or other intelligent features. These dependencies can reduce the need to build every capability internally, but they also introduce risks related to provider availability, pricing changes, model updates, data handling policies, and service discontinuation. A change in an external service can affect the application’s functionality even when its own code remains unchanged.

Third-party dependencies may also complicate incident investigation and accountability. Businesses might have limited visibility into how an external model was trained, how its behavior changes between versions, or how submitted data is processed. If a provider changes its retention policy, introduces a new usage restriction, or retires an API, the business may need to revise its data practices or modify application features.

Intellectual property concerns can arise when AI-generated content incorporates protected material, when users submit copyrighted assets for processing, or when a business uses models under licensing terms that restrict particular commercial applications. The extent of these risks depends on the provider’s terms, the content involved, and the applicable law. Businesses should review provider agreements, understand data usage provisions, and consider contingency plans for critical AI services.

How AI Risks Affect Mobile App Users and Businesses?

The consequences of AI risks depend on an application’s purpose, the sensitivity of the information it handles, and the level of trust users place in its outputs. A minor recommendation error may have little impact, while a privacy breach, biased decision, or misleading response in a high-impact application can cause substantial harm. Businesses need to consider both immediate technical consequences and the longer-term effects on users, operations, and customer relationships.

The impact of AI-related risks generally falls into the following areas:

Impact on Mobile App Users

AI-related failures can affect users’ privacy, decision-making, and confidence in an application. Common consequences include:

• Loss of privacy: Exposure or misuse of personal information can reduce users’ control over how their data is collected and processed.
• Misleading information: Inaccurate AI-generated responses or recommendations can lead users to make poorly informed decisions.
• Unfair treatment: Biased algorithms can produce unequal recommendations, classifications, or outcomes for different groups of users.
• Reduced trust: Unexpected AI behavior, unclear data practices, or repeated errors can discourage users from engaging with AI-powered features.
• Potential harm: Errors in healthcare, financial, or other high-impact applications can have consequences beyond a poor user experience.

Impact on Businesses

For businesses, AI risks can affect financial performance, regulatory obligations, operational continuity, and brand reputation. The potential consequences include:

• Financial losses: Incident response, legal expenses, compensation, and unexpected AI service usage can increase business costs.
• Reputational damage: Publicized privacy violations, biased decisions, or unreliable AI outputs can weaken customer confidence.
• Regulatory consequences: Mishandling personal data or failing to meet applicable AI-related obligations can lead to investigations, penalties, or corrective requirements.
• Operational disruption: Model failures, external service outages, and security incidents can interrupt important app features and business processes.
• Higher support and maintenance costs: Employees may need to correct AI-generated errors, investigate complaints, and address recurring technical problems.
• Reduced user retention: Repeated errors or concerns about data handling can discourage customers from continuing to use the application.

How AI Risks Can Compound

AI-related problems do not always occur in isolation. A single incident can create several connected consequences for users and businesses. For example, if an AI assistant exposes customer information, the incident may trigger a security investigation, increase remediation expenses, damage customer confidence, and raise questions about regulatory compliance.

Similarly, biased recommendations can lead to unfair user experiences, complaints, reduced engagement, and potential legal or reputational concerns. The overall impact depends on the nature of the failure, the number of users affected, the sensitivity of the information involved, and the effectiveness of the business’s response.

Businesses should therefore evaluate AI risks according to their application’s intended purpose and the potential consequences of failure, rather than focusing only on the sophistication of the underlying technology. Understanding how these risks affect users and business operations provides a clearer basis for making informed AI adoption decisions and setting realistic expectations for mobile app reliability.

Conclusion

AI introduces new possibilities for mobile applications, but it also creates risks involving data privacy, security, accuracy, fairness, compliance, and operational reliability. The severity of these risks depends on how AI is integrated, what information it processes, and how much users and business operations rely on its outputs. Understanding these factors helps businesses make informed decisions about where and how to use AI in their mobile products.

For businesses, the challenge is not simply to identify potential AI failures but to understand their consequences for users, operations, and long-term trust. Considering these risks alongside application requirements, technology choices, and ongoing operational costs provides a clearer basis for evaluating AI-powered mobile app initiatives. As AI capabilities continue to evolve, risk awareness will remain an important part of responsible mobile app development.

The post Top AI Risks in Mobile Apps: What Businesses Need to Know appeared first on TopDevelopers.co.

Which humanoid robots are closest to deployment in 2027? Compare Tesla, Figure AI, Agility Robotics, and other leaders in industrial automation.

The post Humanoid Robots in 2027: Tesla, Figure AI, and Agility Race to Deploy appeared first on TechRepublic.

Which humanoid robots are closest to deployment in 2027? Compare Tesla, Figure AI, Agility Robotics, and other leaders in industrial automation.

The post Humanoid Robots in 2027: Tesla, Figure AI, and Agility Race to Deploy appeared first on TechRepublic.

Across a business, any number of AI platforms, systems, and tools may be at work simultaneously—for example, AI-powered customer platforms, conversational assistants, and workflow and process automation tools. Each one is sophisticated, advanced, and quick to respond. But, when set to work, the outcomes and actions that emerge may be wrong, confused, or lacking a proper understanding of the business. Many organizations find themselves in this situation—and it could be one of key reasons they struggle to extract returns on their AI investments.

An MIT Technology Review Insights poll finds that while every organization surveyed is using AI today to some degree, fewer than one in five (16%) are extracting transformative value from their deployments. The largest percentage of respondents (40%) say they are seeing moderate gains in specific tasks or teams, while 21% report meaningful gains, leaving 23% that are seeing minimal returns or none at all.

Companies are mostly still piloting AI use cases (according to 38% of respondents) or using it to assist employees’ work with tools like co-pilots or chat assistants (35%). Just 12% of enterprises have AI agents in production and 3% say they are using AI to do most of the work in key processes.

“There’s a big gap between what the models are capable of and how the models are actually being useful inside organizations,” says Prukalpa Sankar, co-founder of Atlan. “For every real-life intent and purpose, artificial general intelligence is already here. It’s just not really useful in most companies today.”

The missing layer, Sankar says, is context, or how well AI understands a business. A context layer is the architectural infrastructure that sits between an organization’s business systems (for example, systems of record and data, and its AI platforms). It translates enterprise intelligence into governed, machine-usable context for AI agents or models and helps knowledge compound. “Without a context layer for AI, it’s almost like you’re hiring the smartest intern in the world, blindfolding them, and throwing them into your organization,” Sankar says.

Download the report

This content was produced by Insights, MIT Technology Review’s custom content arm, not its editorial staff. It was researched and written by humans, with any AI tools that may have been used limited to production processes under human oversight.

It's almost like Google's got something to prove after taking a break from releasing new frontier AI models earlier this year. Just two weeks after rolling out Gemini 4 Argon, Google has announced Gemini 4 Carbon. This isn't the Flash model equivalent we've been waiting for but, rather, is an improvement on Argon, and like that model, Google is working on a staged release that could take a while.

Stop me if you've heard this one before: Google says its latest AI model reaches a new threshold in coding, agentic tasks, and scientific analysis. It's so capable, in fact, that Google can't let you use it just yet.

Gemini 4 Carbon reportedly makes big gains in cybersecurity applications. Google says the model shows a "significant leap" in vulnerability detection, validation, and remediation. Carbon is also apparently twice as likely to generate code that is free of vulnerabilities versus Argon. Meanwhile, the Chrome team claims that Carbon is 10 times better at generating "high-quality" patches compared to non-Google models. Google also notes that Carbon has increased its score by 89 points on WebDev Arena, now ranking at number four on the leaderboard.

Read full article

Comments

The Netherlands plans 25,000 government job cuts as it explores AI savings. See the 2027 timeline and steps for Dutch and EMEA public-sector IT teams.

The post Netherlands Plans 25,000 Job Cuts as Government Turns to AI appeared first on TechRepublic.

The Netherlands plans 25,000 government job cuts as it explores AI savings. See the 2027 timeline and steps for Dutch and EMEA public-sector IT teams.

The post Netherlands Plans 25,000 Job Cuts as Government Turns to AI appeared first on TechRepublic.

Apple Creator Studio will add AI assistants to Pages, Numbers, and Keynote in November. Check pricing, device requirements, and business account limits.

The post Apple Creator Studio to Add AI Assistants to Pages, Numbers, and Keynote appeared first on TechRepublic.

AI data centers are creating new challenges for power grids. Learn how emerging reliability standards and power technologies could reshape their operations.

The post AI Data Centers Are Straining Power Grids. New Rules Aim to Reduce the Risk appeared first on TechRepublic.

Google’s Genie 4 adds native audio and real-time world editing. Learn about access, US pricing, and the limits teams should assess before running a pilot.

The post Google Unveils Genie 4 With Native Audio and Real-Time World Editing appeared first on TechRepublic.

Samsung partners with Lark Health to offer eligible US seniors AI diabetes prevention coaching through Samsung Health, with Medicare eligibility restrictions.

The post Samsung Health to Offer Free AI Diabetes Prevention Coaching for Seniors appeared first on TechRepublic.

Opera's WebAI brings local AI models to supported browsers, offering developers new options while raising questions about privacy and device requirements.

The post appeared first on TechRepublic.

Apple Creator Studio will add AI assistants to Pages, Numbers, and Keynote in November. Check pricing, device requirements, and business account limits.

The post Apple Creator Studio to Add AI Assistants to Pages, Numbers, and Keynote appeared first on TechRepublic.

AI data centers are creating new challenges for power grids. Learn how emerging reliability standards and power technologies could reshape their operations.

The post AI Data Centers Are Straining Power Grids. New Rules Aim to Reduce the Risk appeared first on TechRepublic.

Google’s Genie 4 adds native audio and real-time world editing. Learn about access, US pricing, and the limits teams should assess before running a pilot.

The post Google Unveils Genie 4 With Native Audio and Real-Time World Editing appeared first on TechRepublic.