SMILE-4-VIP: Smart eMaIl Link domain Extractor to support Visual Impaired People
Current State of Research/Technology
Different definitions of phishing exist. In this post, we consider phishing emails as emails that contain malicious links. Attackers can install malware on the respective device simply by visiting a website through these links, or they can redirect to a phishing website that looks deceptively real and is used by the attacker to harvest login credentials. Phishing emails vary in how easily they can be recognized as such. Examining the URL behind a link is the most reliable indicator of whether it is safe to click on the link or not, even in well-crafted phishing emails. The so-called "Who" part (Domain+TLD) of the URL plays the most crucial role.
Figure 1: The "Who" part as a crucial indicator for detecting malicious links in phishing emails.
There are various approaches to reduce the risk. This includes, initially, the improvement of tools used by email servers to classify emails as phishing. This is crucial to minimize the number of phishing emails reaching users' inboxes. However, since these tools are constantly playing catch-up with new attack strategies, it will not be possible in the future for a tool to identify all phishing emails as such (and simultaneously not render itself unusable due to a high false-positive rate). Phishing emails that the tool doesn't recognize can only be identified by the users themselves.
To precisely support this identification, there are both security awareness approaches that emphasize the importance of checking the URL behind links before clicking (and how to check it), as well as approaches that assist end-users with security interventions displayed "just-in-time-and-place" (as seen in TORPEDO or SMILE) to verify URLs before clicking.
Barrier-free use, especially the support for individuals with severe visual impairment and blindness, plays a very subordinate role in these approaches. It has not been investigated how effective these approaches can be for this group of people. Blind individuals are particularly vulnerable, as emails are read aloud – often at high speed and with the omission of long URLs – and it is not possible to get an immediate overview of whether, for example, the design of an email already appears suspicious.
Existing awareness measures are generally not accessible, nor is the implementation of the content possible for the affected groups, as handling emails and especially links differs. Security interventions in the form of tooltips also do not work when emails are being read aloud, and links are not initially touched, preventing a tooltip from appearing.
This is precisely where SMILE-4-VIP comes into play. It combines two existing approaches (TORPEDO and SMILE) and adapts them to the needs of accessibility.
This is precisely where SMILE-4-VIP comes into play. It combines two existing approaches (TORPEDO and SMILE) and adapts them to the needs of accessibility.
From TORPEDO, SMILE-4-VIP adopts the idea that URLs behind links are categorized into various risk levels: In the case of low risk, a tooltip appears with a green or blue border containing the underlying URL – green if the "Who" part is included in TORPEDO's default known list (Allow-List), and blue if users have previously classified this "Who" part as low risk. The latter is possible either through settings or happens automatically when users have accessed a link to the same "Who" part twice. If TORPEDO detects an unknown risk, the tooltip has a gray border and includes additional information about its verification alongside the underlying URL. TORPEDO supports various special cases, such as resolving short URLs and redirections. In this case, the displayed tooltip contains not the underlying URL but the target URL.
Figure 2: TORPEDO (left low risk and right unknown risk)
The idea behind SMILE is to replace all links contained in the email with SMILE-links. SMILE-links are links where the link text is the 'Who' part of the URL behind the original link. Unlike TORPEDO, relevant information would be read aloud to individuals with severe visual impairment and blindness. However, due to the high speed of reading, this could be easily overlooked. This is especially true for attacks where the phishing 'Who' part is similar to the legitimate 'Who' part (e.g., arnazon.de). Furthermore, all links would need to be checked, as there is no distinction between low risk (can be opened easily) and unknown risk (should be carefully checked).
Figure 3: SMILE (left without and right with SMILE)
Originality of the guiding idea
For individuals with severe visual impairment and especially for blind individuals, it is currently particularly challenging to identify phishing emails. Firstly, there is a lack of phishing awareness measures adapted to their needs. Even if the importance of checking the URL behind the link is known, the problem remains that the rapid reading of the URL behind the link makes it nearly impossible to hear and analyze the 'Who' part. Additionally, due to the rapid reading, small deviations are difficult to discern (e.g., amazon.de versus arnazon.de). Furthermore, it has long been communicated that one should pay attention to spelling errors for phishing detection and that the imposition of time pressure combined with invoices and warnings could be an indication of phishing. Spelling errors are much more challenging to detect when read aloud compared to independent reading. Therefore, even this criterion is not easy to apply for people who have emails read to them. As email is now used for communication with many online services, legitimate invoices and warnings are not uncommon. To be on the safe side and avoid falling victim to a phishing attack, individuals with severe visual impairment or blindness would have to delete emails with time pressure in combination with invoices or warnings. This not only creates significant uncertainty but also causes these individuals to miss important cues, invoices, warnings, or other essential tasks that are exclusively handled electronically.
"If I don't make progress here, the risk is taken, and it's preferred to delete a bit too much, just to be safe." (Statement of a blind student)
Due to significant uncertainty and the lack of experience in dealing with phishing emails, a procedure that enriches emails, for example, in the subject line with information and optionally an additional alert tone, is helpful for any form of highlighting or filtering. An additional alert tone fulfills a crucial criterion for accessibility, the Two-Sensors Principle. It states that a signal should always be perceived with two sensors, such as vibration and simultaneous sound signal at a traffic light. Such a method of alerting to phishing emails has not been implemented in any email client so far.
Therefore, SMILE-4-VIP aims to assist individuals with severe visual impairment or blindness in detecting phishing emails. It focuses on phishing emails that were not identified by the security checks of the email server (the false negatives) and are therefore delivered to the inbox. Specifically, individuals with severe visual impairment and particularly blind individuals should be protected from clicking on the dangerous links in phishing emails that the email server "mistakenly" delivered. At the same time, the usability of handling emails for individuals with severe visual impairment or blindness should be influenced as little as possible. For example, the screen reader (the reading software for individuals with severe visual impairment or blindness) that reads emails should not be modified so that SMILE-4-VIP works independently of the screen reader in use. Accordingly, the goal of SMILE-4-VIP is to make as few changes as possible to the email clients or the emails themselves, so that reading strategies remain as consistent as possible and the effort for detecting phishing emails or dangerous links remains manageable.
SMILE-4-VIP is an extension for email clients. When downloading emails, all URLs behind the links are examined by SMILE-4-VIP, and a risk level is assigned to the email. Depending on the risk level, the email is either not modified at all or a security notice is added to the subject line and possibly the email body is adjusted. If the email body is adjusted, this is done similarly to SMILE4, but due to the previous risk check, it occurs in much fewer cases than with SMILE4.
SMILE-4-VIP modifies the subject of the email in the email client, as the subject and sender are the information that individuals with severe visual impairment or blindness have read aloud to decide whether they want to examine the email further (and thus have it read aloud) or not. To ensure that this notification is reliably perceived, the Two-Sensors Principle is also incorporated, and an acoustic alert using Auditory Icon or Earcon is played, which sounds when the email is selected.
Figure 4: Email Overview with SMILE-4-VIP. Note: The account was created to demonstrate different cases. This does not represent the distribution of cases in real inboxes. The phishing emails contain spoofed senders, so the emails can only be classified based on the URLs behind the links.
"Anything that warns is good. In the subject line, it would be really practical because I turn off most of the client's notifications as they always disrupt reading." (Statement from a blind student)
"I find the phishing warning in the subject line to be a great help. It works much better than message windows from other clients that I know. I generally turn off message windows wherever possible because they snatch away focus, such as the Braille line under the fingers, when they pop up." (Statement from a blind KIT staff member)
The risk levels of SMILE-4-VIP are the levels defined by TORPEDO. In SMILE-4-VIP, the risk level refers to the entire email, whereas TORPEDO defines the risk for each link in the email separately. SMILE-4-VIP assigns the risk level of the email to the most critical level among the links contained in the email. This results in the following cases of how emails are classified and treated by SMILE-4-VIP:
- Low Risk: All links either point to web servers on the predefined allow-list or to web servers that the user has already visited twice in the past (since using SMILE-4-VIP). In this case, the email remains unchanged, meaning no notice is added to the subject, and the email body is not altered (see the second and fourth emails in Figure 4).
- Unknown Risk: At least one link points to a web server not on the predefined allow-list and has not been visited by the user before. In the treatment and representation of this level, SMILE-4-VIP distinguishes two sub-cases:
- The email contains only links to a web server for which SMILE-4-VIP cannot determine the risk. In this case, the subject is adjusted. In addition to the auditory alert, the following text is inserted before the actual subject: "Security Alert: Link to an unknown server <Who part of the URLs behind the links>: " (see the first and third emails in Figure 4).
- The email contains links to different web servers, for at least two of which SMILE-4-VIP cannot determine the risk. In this case, both the subject and body are adjusted. In addition to the auditory alert, the following text is inserted before the actual subject: "Security Alert: Links to multiple unknown servers: ". The email body is adjusted similarly to SMILE, meaning the text directly includes information about the 'Who' parts of the URLs behind the links. In this case, capturing the content of the email is only slightly more time-consuming. It is ensured that the 'Who' part of the URL is read aloud before clicking on the corresponding link.
Figure 5: Example of an email (excerpt) containing links to different webservers, for which SMILE-4-VIP could not determine the risk for at least two.
An email classified as "unknown risk" in SMILE-4-VIP can, like TORPEDO, either indicate a low risk (see the first email in Figure 4) or be a phishing email (see the third email in Figure 4). The user makes this decision based on the information provided by SMILE-4-VIP, as only users have the context and can make such determinations.
SMILE-4-VIP also builds on TORPEDO functionality regarding the investigation of URLs behind links, especially regarding redirection URLs: SMILE-4-VIP also determines the target URLs for known redirect URLs and short URLs without accessing the webpage—thus avoiding any risk to the user. The risk level of the email incorporates the risk level of the target URL. In the case of an unknown risk for the target URL, SMILE-4-VIP displays the who part (domain part) of the target URL in the subject or uses it in the body.
The text displayed in the subject is configurable and can be shortened by users if desired (e.g., to "Server host123.de unknown"). Like TORPEDO, the Allow List can be adjusted, and entries from the user's (blue) list can be removed or added through settings. For instance, a company might customize the lists for its employees, classifying commonly used domains or servers as low risk, significantly reducing the number of emails initially classified as unknown. This improves both efficiency and user experience.
Before using or activating the email client extension, users receive a brief introduction to the functionality of SMILE-4-VIP. This is particularly important to explain to users (1) that the security notice means technical security checks cannot assess the risk of clicking the link, and users must decide themselves, (2) that emails with a security notice can be legitimate and harmless, but could also be phishing emails, and (3) how to determine whether an email with a security notice is a phishing email or not. It is crucial for users to understand this concept, especially since communicating an "unknown" risk is not common in security interventions. Therefore, this introduction aims to prevent emails with the "security notice" addition from being immediately deleted, as there is a risk of deleting important legitimate emails. How to best present this information for people with severe visual impairment and blindness is part of ongoing research.
The following video provides an impression of the working techniques, especially for blind individuals, and exemplifies how SMILE-4-VIP functions in action:
The idea behind SMILE-4-VIP is so fundamental and important that it can be extended to many other areas and applications, offering significant added value to affected individuals in dealing with digital platforms in various fields. Cybercriminals embed phishing links not only in emails but also in social media posts and text messages (SMS, iMessages). Corresponding extensions of open-source apps or add-ons for web browsers, using the same approach, would significantly reduce the risk in these scenarios. Additionally, such extensions also assist people with limited text comprehension, cognitive impairment, or other sensory limitations since SMILE-4-VIP focuses attention on potential dangers through the adjusted subject, thus emphasizing critical parts during reading.
Societal Relevance and International Potential
The widespread adoption of digital technologies, particularly during the COVID-19 pandemic, has demonstrated their pivotal role in overcoming physical barriers and facilitating various aspects of daily life. From remote work to virtual gatherings with friends and family, digital connectivity has proven instrumental. However, as technology evolves, so do the associated risks, especially in the context of cybercrime.
The prevalence of cyber threats, particularly in online communication such as email, poses significant challenges. Phishing attacks, often executed through deceptive email links, can lead to substantial damages for both individuals and organizations. The financial toll of such attacks is substantial, with the U.S. alone reporting $54 million in damages in 2020, according to the IC3 Annual Report.
People with severe visual impairments and blindness face heightened vulnerability in the digital landscape, given the way they interact with emails and links. Therefore, there is a critical need for organizations and companies to prioritize support for this user group in detecting phishing emails. A single interaction with a phishing email by an employee could lead to severe consequences for an organization.
SMILE-4-VIP serves as a simple yet effective measure to address this challenge. It minimally influences email interaction while providing substantial value in an inclusive context. Individuals with severe visual impairments and blindness receive crucial information about potential risks through the subject line while the email is being read. This early notification empowers users to decide whether an email poses an "unknown" risk, facilitating their ability to discern phishing attempts.
By aiding organizations and companies in enhancing security measures for individuals with visual impairments, SMILE-4-VIP aligns with legal requirements for accessibility, such as the EU Directive 2016/2102 and national regulations like the "new" BITV 2.0 in Germany. Ensuring barrier-free access to information security measures becomes imperative in meeting these legal obligations and fostering an inclusive digital environment.
Additionally, SMILE-4-VIP's design and functionality suggest its potential applicability beyond users with visual impairments. Successful phishing attacks often exploit emotions and other psychological factors. SMILE-4-VIP's early notification system provides a preventive measure against such tactics, reducing the risk for individuals without visual impairments as well. If research can demonstrate a significant improvement in phishing detection for users without visual impairments using SMILE-4-VIP, its overall utility and impact would be greatly amplified.
In summary, SMILE-4-VIP not only addresses a critical need for individuals with visual impairments but also contributes to a more secure digital landscape for everyone, making it a valuable tool in the fight against phishing attacks.
Contact: