A Cellular Telephone-Based Application for Skin-Grading to Support Cosmetic Sales

Door-to-door sales is one of the most popular sales strategies in Japan. Salespeople visit a customer's home, promote new products, and help the customer select suitable products through face-to-face communication. We can regard this strategy as one of customer-relationship management (CRM) (Goldenberg 2002). CRM is not the product-oriented concept of "good products can be sold well" but the customer-oriented concept of "only products that a customer desires can be sold." The most important point of CRM is to maintain good customers, and the periodic communication door-to-door salespeople have with customers is an effective way for a company to do this. However, door-to-door sales volume is decreasing because of the many online shops from which one can get detailed product information without talking with salespeople.

The target users for our system are associated with cosmetics companies. Basically, cosmetics companies have failed to keep up with information technology (IT), so we designed a door-to-door sales-support system that helps sales-people by employing IT and AI technologies. This sales-support system is a CRM tool for door-to-door sales of cosmetics based on Skin-Expert, a skin-image grading system that provides analysis, diagnosis, and recommendations. Skin-Expert analyzes the current grade (quality) of a customer's skin from a picture of the skin. Several parameters are extracted by image processing, and the skin grading is done by rules generated through data mining from a baseline of grades given by human skin-care experts.

Communication with the Skin-Expert is through a cellular telephone with a camera, using e-mail software and a Web browser. This type of a cellular telephone is quite popular in Japan. Thus our system does not require that end users have any special hardware. Salespeople take a picture of the customer's skin using the camera in the telephone and send the picture by e-mail to our analysis system (figure 1). The skin-grading system analyzes the picture. The results are made available as a page in HTML format on a customer-accessible Web site. An e-mail is sent when the results are available, usually within minutes. Salespeople check the results by using a Web browser on their cellular telephones. The output provides not only a grading result but also recommendations for the care and cosmetics that are most suitable for the customer.

Our system integrates Web technology, computer analysis, data mining, and an expert system. Although salespeople use only a cellular telephone with very little computing power as the front end, they can take advantage of intelligent services such as computer grading and data mining. The salespeople do not need to think about what is running in the background.

This service is provided through Wisdom-Tex, Inc. (www.wisdomtex.com), a venture company from Tokyo University of Science that provides data-mining services. At the time of this writing, Skin-Expert has been deployed within seven cosmetics companies. In our business model, we intend to charge a minimum rate to make the service affordable for part-time salespeople; we charge only ten dollars a month. This business model makes sense because the largest cosmetics company has more than one hundred thousand door-to-door salespeople, and overall there are about one million such salespeople in Japan.

We have organized this article as follows: the next section provides an outline of our Skin-Expert; the third section explains the skin-image processing of our Skin-Expert; the fourth section describes the data mining used to create the skin grading; the fifth section introduces the expert system that constructs advice about cosmetics suitable for a customer; the sixth section shows the performance of our Skin-Expert; and the final section contains our conclusions.

Skin-Expert

Skin grading from skin images is common in most cosmetics companies in Japan. They have used a special camera to take close-up images of the skin. These images are then judged by human experts who determine the skin grade from the image and who then provide individualized product recommendations.

Our Skin-Expert automatically diagnoses skin images using rules extracted by data mining. Moreover, we can use a cellular telephone with a camera to obtain and transmit the images (figure 1). About 50 percent of Japanese people now have a cellular telephone, and most cellular-telephone manufacturers now produce a cellular telephone with a camera, so sending pictures and movies by e-mail has recently become common in Japan.

Cellular-telephone technology has been progressing rapidly. Some cellular telephones now have close-up lenses (with up to seven times magnification), zooming functions (up to ten times), and autofocus. For example, the Casio A5403CA has a 2-megapixel charge-coupled device (CCD), autofocus, and ten times zoom. When we first started deploying our system, we had to provide a loupe (magnifying glass) over the standard cellular-telephone camera lens for taking close-up images of the skin (Hiraishi and Mizoguchi 2003). Now, however, we can take close-up images for analysis by Skin-Expert using only the standard camera on current cellular telephones.

Using the close-up mode of the cellular telephone camera (figure Ia), we take a skin image from about 15 centimeters, as depicted in figures Ib and Ic. We can take the skin image easily by fixing the camera on the customer's shoulder. (It was a little hard to take a clear skin image by using a loupe because the focal length was very short. Now it is easier to take the skin image.) The skin image is sent to our system by using e-mail software on the cellular telephone (figure Id). An e-mail is sent back when the results are available, usually within minutes. The return message contains the URL of the results page (figure Ie). When we click the URL, the browser on the cellular telephone starts, and we can check the grading result and recommendations as illustrated in figures If and Ig.

The most significant feature of our system is that it does not require that end users have computers and special camera devices to perform automatic skin grading. Almost all salespeople in cosmetics companies work part time, and almost all salespeople are very inexperienced with computers. For these reasons, our approach in designing the Skin-Expert system-not forcing salespeople to buy any special hardware but using the cellular telephone they ordinarily use-is well suited for this application.

Figure 2 shows homepages of our Skin-Expert as the user interface on the cellular telephone. The upper-left page is the member's page: there are links for checking the results, referring to past results, requesting skin grading, inputting user information (user name, e-mail address, and so on), and modifying the system configuration (user password, time-out, and so on).

The upper-right page is a member's results page. Seven factors represent the current status of the skin: the number of pores, the depth of the pores (this is reflected in the degree of darkness of skin), the thickness of the wrinkles (we can regard this as the degree of tension of the skin), the depth of the wrinkles (we can regard this as the degree of dryness of the skin), the clearness of a wrinkle, the clearness of the skin texture, and the cumulative skin grade based on these parameters. The evaluation is also moderated by parameters such as the person's gender and age.

The lower-left page of figure 2 is the skin-grade page. The skin grade is represented as a number of stars. Five stars represent the best skin status and one star the worst. We can check the balance of seven factors from this page, represented in a radar chart (figure If). We can see the recommended cosmetics that are linked from each results page on the lower-right page.

Architecture of Skin-Expert

Figure 3 shows the architecture of Skin-Expert. We attach a skin image (in JPEG format) taken with the cellular telephone to an e-mail message and add information such as the customer's name, age, and normal sleeping hours in the mail body. This is processed by our server.

We are using a GRID computer (Foster and Kesselman 2003) to cope with many requests. In the current configuration, it takes about 1 second of processor time to analyze one skin image. If one hundred thousand requests arrived at the same time (this is a little hard to consider, even though one company has more than one hundred thousand salespeople) and they were processed sequentially, it would take more than twenty-seven hours to finish all the requests. So, we need a lot of accessible computing power for door-to-door sales support in order not to make customers wait.

The server software running on the master host of the GRID computer checks the mailbox periodically. When a mail request arrives in the mailbox, the server software reads the attached image and information in the mail body. The skin-grading software is the main part of our Skin-Expert system. The software analyzes skin images by image processing and grades the skin using the rules generated by data mining; it also generates some recommendations. We present the details of these functions in the following sections.

The skin-grading task is distributed by the Grid Engine (Sun Microsystems 2002) and is executed by the skin-grading software running on an execution host. After finishing the skin grading, the skin-grading software puts the results on the Web server as an HTML file, and the skin-grading software sends an e-mail message to the cellular telephone to report the completion of the skin grading. The URL to the results page is included in the e-mail message, so customers can check the results and the recommended cosmetics using the cellular telephone's Web browser.

Skin-Image Processing

In our Skin-Expert, the original color image (240 320 pixels) is converted to a gray image, an edge image, a noise-reduced image, and a line image like the ones shown in figure 4. Several parameters are extracted during the image processing.

The extracted parameters are (1) the number of intersections; (2) the image depth at the intersections; (3) the line thickness; (4) the image depth of lines; and (4) the strength of line direction.

The number of the intersections is the number of the intersections between each line on the line image. We can interpret intersections as pores. In the case of figure 4, the number of intersections is 84.

The image depth at the intersections is the average color depth (0-255) of the intersections in the line image. It can be interpreted as the normalized pore depth. The coordinates of the intersections can also be obtained from the line image. The intersections are located in the gray image, and the color depth at that point of the image is closely correlated with the pore depth. The parameter is 126 for figure 4.

The line thickness indicates the thickness of wrinkles. The black part of the noise-reduced image of figure 4 can be defined as the wrinkle part. The line thickness is calculated by dividing the sum of the black pixels in the noise-reduced image by the sum of the black pixels in the line image. In the case of figure 4, it is 4.04.

The image depth of lines correlates with the depth of wrinkles. It is computed in the same way as the image depth of the intersection. It is represented as the color depth related to the gray image by using the coordinates of the line in the line image. It is 123 in figure 4.

Skin in which we can see the wrinkle direction is not in good condition. In the case of figure 4, we can see the wrinkle direction from upper right to lower left. We define the strength of line direction for skin grading.

First, the standard deviation of the pixel value on the 36 lines (every 5 degrees) from the center is calculated on the gray image (figure 5a). Here, the pixel value on the line is the average of pixels on the perpendicular line (figure 5b). Then the strength of the line direction is represented as the standard deviation of the standard deviation of each line. If the direction of the wrinkle is clear, the standard deviation of the line across the wrinkles is big (figure 5c). In contrast, if the direction of the wrinkles is not clear, the standard deviation of the line along the wrinkles is small (figure 5d), so the whole standard deviation becomes small. The strength of figure 4 is 11.8.

Data Mining for Skin-Expert

We have about ten thousand sample skin images that have been gathered through collaboration with cosmetics companies and that have been graded by human experts. The parameters previously described have been extracted from these baseline cases, and we used data mining (as we will describe) to extract rules for skin grading to try to come close to the human performance. There are three other parameters of skin grade: (1) clearness of wrinkle, (2) clearness of texture, and (3) total evaluation.

The clearness of wrinkle of the skin is evaluated at one of three levels (three stars, two stars, and one star). Skin that has a clear texture is good skin. Clearness of texture is evaluated at three levels (three stars, two stars, and one star). Considered with the preceding two grades, the total evaluation of the skin is determined as being at one of five levels (five stars, four stars, three stars, two stars, and one star) by the experts, and also by our program.

We used inductive-logic programming (ILP) (Muggleton 1995) for doing our machine learning. ILP is a machine-learning technique based on first-order logic. It can extract relational rules among several attributes. Figure 6 is an example rule for skin grading extracted by our constraint version of ILP (Mizoguchi et al. 1995).

The grade in line 1 of figure 6 indicates that this rule is for the three-stars grade. This rule means that "If the sex is female (line 2), the age is 40s (line 3), the number of the intersections NI is between 80 and 90 (line 4), the image depth of the intersections DI is between 120 and 130 (line 5), the line thickness LT is between 3.1 and 5.1 (line 6), the image depth of the lines LD is between 20 and 130 (line T), and the strength of the line direction LS is between 9.1 and 12.5 (line 8), then the grade is three-stars skin."

Table 1 shows the comparison between the grade judged by human expert and the grade evaluated by computer. Perfect accuracy is achieved 38 percent of the time, and a mistake of just one grade happens 42 percent of the time. A two-grade mistake occurs 13 percent of the time, a three-grade mistake occurs 7 percent of the time, and there were no four-grade mistakes. Cosmetics companies have indicated that a one-grade mistake is acceptable. Thus we achieve sufficient accuracy 80 percent of the time.

In addition to rules for the skin grading, the rules related to cosmetics are also extracted by ILP. The rule in figure six represents which cosmetics are suitable for a customer.

The rule in figure 7 contains the customer attribute (favor, age, sex, grade) and the product attribute (maker, component, effect). This rule means that "The manufacturer of the cosmetics is maker_a (line 2); the ingredient is olive oil (line 3); the cosmetics are effective for dry skin (line 4) and good for the customer whose favorite manufacturer is maker_a (line 6), whose sex is female (line 7), and whose age is 40s (line 8); and the grade is three stars."

This type of rule is generated by using the information about the recommendations that were given by the experts and that were accepted by the cosmetics customers. The rules take into account customer and cosmetics information and the actual effectiveness of the rule (shown in terms of the good jor clause in line 5). This is typical of rules used by the expert system to provide advice to customers.

Generating Recommendations

After skin grading, our Skin-Expert can give recommendations about care and cosmetics suitable for a customer. We use the expert system to make recommendations. The expert system can propose the most suitable product for a customer through question-and-answer dialog. If there are some similar products, the system can show them in the order of the "certainty factor" (Shortliffe 1976, Randall, p. 982), which one can regard as the strength of the recommendation.

Figure 8 shows homepages that advise about recommended cosmetics. A customer answers questions shown in the left page. The recommended cosmetics are shown on the right page. The cosmetic with the highest certainty factor is put on the top. Other cosmetics are put in the order of the certainty factor. The parameters from image processing and skin grading and the user profile's age and sex information, which the Skin-Expert system already has, are input automatically without asking questions.

The following processes are repeated in the expert system:

Step 1: Question

Questions about the customer are asked. These relate to age, sex, sleep time, lifestyle, cosmetics the customer usually uses, their effectiveness, and so on. A question is described in the following form:

<question name=cosmetics initial=no type=single>

<prompt>

Which cosmetics are you using?

</prompt>

<condition>

<fact qname=maker item=maker_a active=true />

This represents a question about the cosmetics that a customer usually uses. The initial parameter in the question tag indicates the initial selection. The initial value of no indicates that there is no initial selection. If the initial value is other, a product has already been selected when this question is made. The type parameter indicates the question type. If the type value is single, the customer can select only one item. In contrast, if the type value is multiple, the customer can select several items.

The prompt tag represents a query for a customer. The condition tag indicates the condition that causes this question to be made. In this case, the question is asked if the item's maker_a is selected (active=true) in the question about the cosmetics that a customer usually uses. Finally, the candidate items are described in the candidate tag. The item tag contains the description of the item. In this case, it is the name of the cosmetics, which is displayed in figure 8 on the left.

The exact meaning of this hypothesis is that the value white is the superior to the value premium. Our system understands that the premium is similar to the white. So, if the certainty factor of the premium is changed, the certainty factor of the white is changed as well. Furthermore, the hypothesis tag contains several items to display this recommendation. For example, the item named html contains the HTML description, and the item named image contains the image of the cosmetics product. The recommended cosmetics are thus displayed on the page as in figure 8 on the right.

The expert system of our Skin-Expert repeats the processes of asking a question, applying rules, and changing certainty factors. When all questions have been answered, some recommendations are displayed in the order of certainty factor as indicated by the description in the hypothesis tag.

Here, the grade of the customer's skin as determined through our Skin-Expert and the relation between the customer's grade and cosmetics, generated by ILP, are used in the rules of expert system. So, this expert system contains the computer grading and data mining.

Performance of Skin-Expert

Our Skin-Expert consists of a mail server to receive skin-grading requests, a Web server to access the results, and a Grid Engine to distribute the skin-grading task (see figure 3).

We tested the performance of our Skin-Expert in our GRID computing environment (figure 9) in the office of WisdomTex Inc., which is a IDK apartment. We constructed a 100 megabyte per second optic fiber network (KD-DI) in the office. The Grid Engine (Sun Grid Engine 5.3) runs on the Sun Fire V880 system (Sun UltraSPARCIII 750 MHz CPU 4, 8 giga-bytes of memory, SolarisS), and there are sixteen Dell OptiPIex SX270 systems (Intel Pentium 4 3.2 GHz CPU, 2 gigabytes of memory, RedHat 8.0) to cope with the skin-grading task. These machines are on the bookshelf as in figure 9 on the right. One mail server (Sendmail 8.12) is running on Sun Enterprise 450 (Ultra-SPARCII 450 MHz CPU, 1 gigabyte of memory, SolarisS), and the Web server (Sun One Application Server 7.0) is running on the Sun Fire V880. All machines are connected through the 100 megabyte per second local area network.

Table 2 shows the performance of each part of our Skin-Expert. We started with a situation in which ten thousand request messages had already arrived at the mail server. We recorded the process time from reading the request message to sending the reply message to notify the user of the finished skin grading. Our GRID computer could process sixteen requests per second. Ten to fifteen accesses were available per second for the Web server. This is not just access to the homepage, but access to the Java ServerPages (JSP) program. The mail server could receive only two request messages per second. From these results, only two requests could be processed per second, since the mail server was the bottleneck in our environment. So we have set up a mail server on each machine now. Our Skin-Expert can process sixteen requests per second. Even if one hundred thousand requests arrive at the same time, all requests can be finished in about one and one-half hours.

Conclusions

In this article, we introduced our Skin-Expert, which is an integration of Web technology, computer grading, data mining, and an expert system. Our Skin-Expert analyzes the customer's current skin grade from a picture of the skin. Several parameters are extracted by the image processing, and the skin grading is done by rules generated by data mining. Our system provides not only the skin grade but also gives advice about cosmetics and skin care to customers. Our system shows the expert knowledge of the skin graders and the experience of high-achieving salespeople. It allows beginners to promote cosmetics as experts and to emulate these high achievers. It yields high-level door-to-door sales for cosmetics companies.

In our system, salespeople use a cellular telephone just as the front end; they need not be concerned about computers running in the background. The intelligence is hidden in our system, providing an example of next-generation Web service. The loupe was necessary for our previous service (Hiraishi and Mizoguchi 2003), but the progress of cellular telephones has made it much easier to use our service. Our service thus parallels the progress of small devices such as cellular telephones and the progress of computing power like the GRID computer. In particular, we set up our GRID computer on the bookshelf in our SOHO. The GRID computer is no longer special but can be used for everyday things.

This model can also be applied to other areas such as health care and medical expert systems. We can regard the cellular telephone of our model as a sensor for monitoring the status of the human body. Small devices monitor the current status of our body, and a supercomputer gathers data and evaluates our health status. We can also integrate some genome and drug information for evaluating health. We developed an expert system for evaluating glaucoma more than twenty years ago (Mizoguchi et al. 1997). We are now expanding the system to incorporate our model.

Our current GRID computer is enough for our service. However, we will have to expand it so as not to decrease the service level as the requested services increase. We have set up several types of GRID computing environment. We can use open grid service architecture (OGSA) (Foster and Kesselman 2003) to integrate these environments for more computational power and back up.

We are in contact with more than sixty-eight cosmetics companies. Of these, 40 percent are considering adoption of our Skin-Expert. Avon Products Company, Ltd., has adopted our service, and about five hundred of the company's salespeople began using our service in March 2004. We customized and coordinated our service for them. We are constructing a database of components for cosmetics products to provide knowledge for custom-producing cosmetics for a customer, and we have developed a skin-aging simulation. We are also gathering skin-cancer data and diagnoses to extract an expert skin-cancer-diagnosis system. The analysis of the data will enable us to judge the skin cancer from just the spot of the skin in skin images taken by the camera of a cellular telephone.

Acknowledgments

We would like to thank Wolfgang Gentzsch of Sun Microsystems, Inc. His presentation at ApGrid 2001 held in Tokyo helped us construct our GRID computer in a short time. Tsuyoshi Kaburaki of Avon Products Company, Ltd., gave us many comments from the end-user viewpoint. His comments helped us to make our system very much more useful. Finally, we have to thank Daniel G. Bobrow of the PaIo Alto Research Center (PARC). He has given us many precious comments about our research projects, and they have helped us very much in achieving our success.