Researchers at an Israeli security firm on Tuesday revealed how hackers could turn a generative AI’s “hallucinations” into a nightmare for an organization’s software supply chain.
In a blog post on the Vulcan Cyber website, researchers Bar Lanyado, Ortel Keizman, and Yair Divinsky illustrated how one could exploit false information generated by ChatGPT about open-source software packages to deliver malicious code into a development environment.
They explained that they’ve seen ChatGPT generate URLs, references, and even code libraries and functions that do not actually exist.
If ChatGPT is fabricating code libraries or packages, attackers could use these hallucinations to spread malicious packages without using suspicious and already detectable techniques like typosquatting or masquerading, they noted.
If an attacker can create a package to replace the “fake” packages recommended by ChatGPT, the researchers continued, they might be able to get a victim to download and use it.
The likelihood of that scenario occurring is increasing, they maintained, as more and more developers migrate from traditional online search domains for code solutions, like Stack Overflow, to AI solutions, like ChatGPT.
“The authors are predicting that as generative AI becomes more popular, it will start receiving developer questions that once would go to Stack Overflow,” explained Daniel Kennedy, research director for information security and networking at 451 Research, which is part of S&P Global Market Intelligence, a global market research company.
“The answers to those questions generated by the AI may not be correct or may refer to packages that no longer or never existed,” he told TechNewsWorld. “A bad actor observing that can create a code package in that name to include malicious code and have it continually recommended to developers by the generative AI tool.”
“The researchers at Vulcan took this a step further by prioritizing the most frequently asked questions on Stack Overflow as the ones they would put to the AI, and see where packages that don’t exist were recommended,” he added.
According to the researchers, they queried Stack Overflow to get the most common questions asked about more than 40 subjects and used the first 100 questions for each subject.
Then, they asked ChatGPT, through its API, all the questions they had collected. They used the API to replicate an attacker’s approach to getting as many non-existent package recommendations as possible in the shortest time.
In each answer, they looked for a pattern in the package installation command and extracted the recommended package. They then checked to see if the recommended package existed. If it didn’t, they tried to publish it themselves.
Malicious packages generated with code from ChatGPT have already been spotted on package installers PyPI and npm, noted Henrik Plate, a security researcher at Endor Labs, a dependency management company in Palo Alto, Calif.
“Large language models can also support attackers in the creation of malware variants that implement the same logic but have different form and structure, for example, by distributing malicious code across different functions, changing identifiers, generating fake comments and dead code or comparable techniques,” he told TechNewsWorld.
The problem with software today is that it is not independently written, observed Ira Winkler, chief information security officer at CYE, a global provider of automated software security technologies.
“It is basically kludged together from lots of software that already exists,” he told TechNewsWorld. “This is very efficient, so a developer does not have to write a common function from scratch.”
However, that can result in developers importing code without properly vetting it.
“Users of ChatGPT are receiving instructions to install open-source software packages that can install a malicious package while thinking it is legitimate,” said Jossef Harush, head of software supply chain security at Checkmarx, an application security company in Tel Aviv, Israel.
“Generally speaking,” he told TechNewsWorld, “the culture of copy-paste-execute is dangerous. Doing so blindly from sources like ChatGPT may lead to supply chain attacks, as the Vulcan research team demonstrated.”
Melissa Bischoping, director of endpoint security research at Tanium, a provider of converged endpoint management in Kirkland, Wash., also cautioned about loose use of third-party code.
“You should never download and execute code you don’t understand and haven’t tested by just grabbing it from a random source — such as open source GitHub repos or now ChatGPT recommendations,” she told TechNewsWorld.
“Any code you intend to run should be evaluated for security, and you should have private copies of it,” she advised. “Do not import directly from public repositories, such as those used in the Vulcan attack.”
She added that attacking a supply chain through shared or imported third-party libraries isn’t novel.
“Use of this strategy will continue,” she warned, “and the best defense is to employ secure coding practices and thoroughly test and review code — especially code developed by a third party — intended for use in production environments.”
“Don’t blindly trust every library or package you find on the internet or in a chat with an AI,” she cautioned.
Know the provenance of your code, added Dan Lorenc, CEO and co-founder of Chaingard, a maker of software supply chain security solutions in Seattle.
“Developer authenticity, verified through signed commits and packages, and getting open source artifacts from a source or vendor you can trust are the only real long-term prevention mechanisms on these Sybil-style attacks on open source,” he told TechNewsWorld.
Authenticating code, though, isn’t always easy, noted Bud Broomhead, CEO of Viakoo, a developer of cyber and physical security software solutions in Mountain View, Calif.
“In many types of digital assets — and in IoT/OT devices in particular — firmware still lacks digital signing or other forms of establishing trust, which makes exploits possible,” he told TechNewsWorld.
“We are in the early innings of generative AI being used for both cyber offense and defense. Credit to Vulcan and other organizations that are detecting and alerting on new threats in time for the language learning models to be tuned towards preventing this form of exploit,” he added.
“Remember,” he continued, “it was only a few months ago that I could ask Chat GPT to create a new piece of malware, and it would. Now it takes very specific and directed guidance for it to create it inadvertently. And hopefully, even that approach will soon be prevented by the AI engines.”
John P. Mello Jr. has been an ECT News Network reporter since 2003. His areas of focus include cybersecurity, IT issues, privacy, e-commerce, social media, artificial intelligence, big data and consumer electronics. He has written and edited for numerous publications, including the Boston Business Journal, the Boston Phoenix, Megapixel.Net and Government Security News. Email John.
Please sign in to post or reply to a comment. New users create a free account.
Loading …
Alphabet
Amazon
Apple
Meta Platforms
Microsoft
Nvidia
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/04/Ubuntu-Lunar-Lobster.jpg
https://www.technewsworld.com/story/ubuntu-23-04-lunar-lobster-lands-with-newly-minted-cinnamon-desktop-flavor-177026.html
Ubuntu 23.04 ‘Lunar Lobster’ Lands With Newly Minted Cinnamon Desktop Flavor
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/05/poly-voyager-free-60-earbuds.jpg
https://www.technewsworld.com/story/poly-voyager-60-series-earbuds-provide-a-premium-audio-experience-178290.html
Poly Voyager 60 Series Earbuds Provide a Premium Audio Experience
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/05/circuit-board-cpu-gpu.jpg
https://www.technewsworld.com/story/2024-and-how-the-coming-cpu-war-is-likely-to-play-out-178325.html
2024 and How the Coming CPU War Is Likely To Play Out
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/05/satisfied-computer-user.jpg
https://www.technewsworld.com/story/linux-what-linux-it-just-works-177075.html
Linux? What Linux? It Just Works
https://www.technewsworld.com/wp-content/uploads/sites/3/2022/03/russian-hacker-group.jpg
https://www.technewsworld.com/story/doj-five-eyes-nations-unite-to-dismantle-russian-cyber-espionage-network-178292.html
DOJ, Five Eyes Nations Unite To Dismantle Russian Cyber-Espionage Network
https://www.technewsworld.com/wp-content/uploads/sites/3/2022/02/remote-worker.jpg
https://www.technewsworld.com/story/personal-data-harvesting-and-how-to-reduce-your-digital-footprint-178046.html
Personal Data Harvesting and How To Reduce Your Digital Footprint
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/03/open-source-pen-1.jpg
https://www.technewsworld.com/story/business-conditions-prime-for-more-open-source-contributors-176940.html
Business Conditions Prime for More Open-Source Contributors
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/05/ChatGPT-iPhone.jpg
https://www.technewsworld.com/story/chatgpts-arrival-on-iphone-sparks-reprise-of-privacy-concerns-178333.html
ChatGPT’s Arrival on iPhone Sparks Reprise of Privacy Concerns
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/03/ai-robot.jpg
https://www.technewsworld.com/story/the-ai-revolution-is-at-a-tipping-point-178143.html
The AI Revolution Is at a Tipping Point
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/01/student-computer.jpg
https://www.technewsworld.com/story/edtech-developers-study-game-approach-aces-med-school-testing-curve-177727.html
EdTech Developer’s Study Game Approach Aces Med School Testing Curve
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/05/RSA-Conference-2023.jpg
https://www.technewsworld.com/story/rsa-conference-rebounds-as-business-risks-soar-178281.html
RSA Conference Rebounds as Business Risks Soar
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/05/AMD-Ryzen-7000-series.jpg
https://www.technewsworld.com/story/amd-could-rescue-the-windows-thin-and-light-laptop-market-from-apple-178319.html
AMD Could Rescue the Windows Thin and Light Laptop Market from Apple
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/03/telehealth-provider.jpg
https://www.technewsworld.com/story/digital-health-care-flourishing-despite-legal-logistical-hurdles-177497.html
Digital Health Care Flourishing Despite Legal, Logistical Hurdles
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/05/Telly-TV.jpg
https://www.technewsworld.com/story/telly-offers-half-million-free-55-inch-4k-tvs-advertisers-to-pick-up-tab-178301.html
Telly Offers Half-Million Free 55-Inch 4K TVs, Advertisers To Pick Up Tab
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/01/smart-home-2-story.jpg
https://www.technewsworld.com/story/leverage-the-power-of-data-to-monitor-home-energy-efficiency-178147.html
Leverage the Power of Data To Monitor Home Energy Efficiency
https://www.technewsworld.com/wp-content/uploads/sites/3/2022/07/iot.jpg
https://www.technewsworld.com/story/unresolved-conflicts-slow-esim-upgrade-path-to-better-iot-security-176886.html
Unresolved Conflicts Slow eSIM Upgrade Path to Better IoT Security
https://www.technewsworld.com/wp-content/uploads/sites/3/2022/06/technical-controller.jpg
https://www.technewsworld.com/story/cyber-chiefs-brace-for-major-attacks-in-next-12-months-178283.html
Cyber Chiefs Brace for Major Attacks in Next 12 Months
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/04/cyberwarfare-3.jpg
https://www.technewsworld.com/story/lazarus-hackers-linux-malware-linked-to-3cx-supply-chain-attack-177020.html
Lazarus Hackers’ Linux Malware Linked to 3CX Supply-Chain Attack
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/05/smartphone-texting.jpg
https://www.technewsworld.com/story/mobile-shopping-apps-linked-to-digital-wallets-a-loyalty-driving-duo-177650.html
Mobile Shopping Apps Linked to Digital Wallets: A Loyalty-Driving Duo
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/04/computer-dual-monitors.jpg
https://www.technewsworld.com/story/new-distro-makes-running-arch-linux-very-cachy-176997.html
New Distro Makes Running Arch Linux Very ‘Cachy’
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/04/ai-cybersecurity-lock.jpg
https://www.technewsworld.com/story/researchers-instantly-crack-simple-passwords-with-ai-178233.html
Researchers Instantly Crack Simple Passwords With AI
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/01/AnkerWork-SR500-Speakerphone.jpg
https://www.technewsworld.com/story/ankerwork-sr500-speakerphone-near-nirvana-for-pc-use-phones-heck-no-177679.html
AnkerWork SR500 Speakerphone: Near Nirvana for PC Use, Phones Heck No
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/03/online-psychology.jpg
https://www.technewsworld.com/story/is-chatgpt-smart-enough-to-practice-mental-health-therapy-178064.html
Is ChatGPT Smart Enough To Practice Mental Health Therapy?
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/03/Bard-wait-list.jpg
https://www.technewsworld.com/story/google-invites-public-to-test-drive-its-ai-chatbot-bard-178066.html
Google Invites Public To Test Drive Its AI Chatbot Bard
https://www.technewsworld.com/wp-content/uploads/sites/3/2022/12/server-racks-data-center.jpg
https://www.technewsworld.com/story/are-mainframes-an-indicator-of-banking-reliability-178260.html
Are Mainframes an Indicator of Banking Reliability?
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/05/Pixel-Fold.jpg
https://www.technewsworld.com/story/google-taking-pre-orders-for-its-first-foldable-phone-178309.html
Google Taking Pre-Orders for Its First Foldable Phone
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/04/Silicon-Valley-Bank.jpg
https://www.technewsworld.com/story/social-media-fueled-the-run-on-silicon-valley-bank-study-178230.html
Social Media Fueled the Run on Silicon Valley Bank: Study
https://www.technewsworld.com/wp-content/uploads/sites/3/2022/08/Space-BACN.jpg
https://www.technewsworld.com/story/darpa-moves-forward-with-project-to-revolutionize-satellite-communication-177003.html
DARPA Moves Forward With Project To Revolutionize Satellite Communication
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/04/vr-female.jpg
https://www.technewsworld.com/story/gen-ai-and-ar-vr-unintended-consequences-unproven-mainstream-appeal-178156.html
Gen AI and AR/VR: Unintended Consequences, Unproven Mainstream Appeal
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/04/phone-charging-station.jpg
https://www.technewsworld.com/story/fbi-issues-warning-about-juice-jacking-at-public-usb-charging-stations-178151.html
FBI Issues Warning About ‘Juice Jacking’ at Public USB Charging Stations
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/03/AI-chip.jpg
https://www.technewsworld.com/story/3-big-generative-ai-problems-yet-to-be-addressed-178213.html
3 Big Generative AI Problems Yet To Be Addressed
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/01/legal-gavel.jpg
https://www.technewsworld.com/story/meta-lowers-legal-hammer-on-law-enforcement-data-scraper-177669.html
Meta Lowers Legal Hammer on Law Enforcement Data Scraper
https://www.technewsworld.com/wp-content/uploads/sites/3/2022/07/evs-charging.jpg
https://www.technewsworld.com/story/study-finds-ev-battery-replacement-rare-most-covered-by-warranty-178257.html
Study Finds EV Battery Replacement Rare, Most Covered by Warranty
https://www.technewsworld.com/wp-content/uploads/sites/3/2022/08/metaverse-future.jpg
https://www.technewsworld.com/story/why-nvidia-is-winning-the-race-to-dominate-the-metaverse-178079.html
Why Nvidia Is Winning the Race To Dominate the Metaverse
https://www.technewsworld.com/wp-content/uploads/sites/3/2023/04/HP-Amplify.jpg
https://www.technewsworld.com/story/hp-affirms-better-together-at-its-amplify-event-178127.html
HP Affirms ‘Better Together’ at Its Amplify Event
https://www.technewsworld.com/wp-content/uploads/sites/3/2022/11/female-soldier-soluting.jpg
https://www.technewsworld.com/story/female-army-veteran-uses-tech-to-help-create-a-better-future-177372.html
Female Army Veteran Uses Tech To Help Create a Better Future
Copyright 1998-2023 ECT News Network, Inc. All Rights Reserved.
Enter your Username and Password to sign in.
