LogInLumirithmic's CTO and Chief Scientist Dr. Abhijeet Ghosh told us about the company's facial capture technology, explained how the company managed to achieve high-quality scans that do not cost thousands of dollars per each, and shared what allows the tech to capture multiple facial expressions within a few minutes.
Lumirithmic
80.lv: Please, introduce yourself and your team. Could you give us a little bit of background about who makes up your team? Where did you study? How did you join the company? What projects did you previously work on? Which of them are you most proud of?
Lumirithmic: Lumirithmic was co-founded by Dr. Abhijeet Ghosh and Gaurav Chawla.
Dr. Ghosh is the CTO/Chief Scientist, currently on sabbatical leave from Imperial College London where he’s a professor in the Department of Computing and leads the Realistic Graphics & Imaging Group. He also holds a PhD in Computer Science from The University of British Columbia and was the recipient of the Alain Fournier Doctoral Dissertation Award, his research has been recognized with a prestigious Royal Society Wolfson Research Merit Awards and an EPSRC Early Career Fellowship.
Prior to Lumirithmic, Dr. Ghosh worked as a Research Assistant Professor at the USC Institute for Creative Technologies where he worked on the development of LightStage technology which was first used in the movie Avatar. His world-leading expertise in appearance capture and modeling has led to technical and consulting roles for companies including Weta Digital, Disney, Avon, and L’Oréal.
Gaurav Chawla is the CEO of Lumirithmic with an extensive 20-year background as a tech/product specialist and deep tech entrepreneur, with specific expertise in product build and commercialization of B2B Enterprise and deep tech solutions. Gaurav also serves as the Entrepreneur in Residence (EIR) at Imperial College London. A graduate of the Indian Institute of Technology (IT, Delhi), Gaurav also holds an MS (Engineering), MS (Mathematical Finance) and MBA (Finance, Accounting) from Georgia Tech, as well as LLB (the equivalent of Juris Doctor) in Intellectual Property Law from the University of London. He is also a CFA and FRM charter holder.
His background also includes engineering, fintech, data science, and computer science which he also held leading technical/product roles at GE Capital, RBS, Morgan Stanley, and IHS Markit. He has co-authored several publications and is also a co-inventor of multiple patents.
Lumirithmic's Facial Capture Technology
80.lv: Could you tell us about your facial capture technology? As far as we know it can provide high-quality facial scans in a matter of minutes. What are the main challenges when creating realistic avatars in such a short period of time?
Dr. Abhijeet Ghosh: Our 3D facial appearance capture technology produces Hollywood-quality facial scans through portable desktop products, enabling 3D scanning at scale for industries including, but not limited to the metaverse, film, gaming, ad-tech, beauty-tech, and skincare. The portable desktop has the ability to successfully scan faces and skin types of all genders and ethnicities at the highest quality in seconds.
The main challenge for creating realistic avatars is accurate to capture both facial shape (geometry) and appearance. Our proprietary capture process and processing algorithms achieve this with our portable desktop products, which is a breakthrough for 3D capture.
Main Limitations
80.lv: What are the main limitations when working with such tech these days? What should be improved?
Dr. Abhijeet Ghosh: In order to achieve realistic facial renderings, traditional 3D capture technology including large cameras and custom lighting rigs was relied upon. With that came complex and expensive systems that could cost thousands of dollars per scan. Lumirithmic has taken a different approach by creating a more scalable and accessible solution. Now, with our 3D facial appearance capture technology, customers will receive richer, robust scans that can be accessed anywhere in the world for a much smaller cost.
The key challenge which we overcame is high quality with accessibility.
Emotion Recognition
80.lv: Emotion recognition could be a challenge as face animation also depends on the way you’re speaking: for instance, laughing, yelling, or whispering could influence facial expressions. How do you cope with these difficulties?
Dr. Abhijeet Ghosh: Our capture process is optimized for speed, and our desktop products can acquire a face in any expression within a couple of seconds. This allows capturing a subject in multiple facial expressions within a few minutes. Acquiring a face in multiple expressions is a standard process in the CGI industry for building facial animation rigs. Furthermore, our dynamic performance capture system enables video-rate direct 3D capture of a subject’s facial performance, which has the advantage of acquiring even subtle facial, lip, and/or eye movements during a performance.
Available and Cost-Effective Product
80.lv: You’ve mentioned that your goal is to "create a much more widely available and cost-effective product for 3D facial appearance capture." What is your approach here? How do you plan to achieve that?
Dr. Abhijeet Ghosh: As mentioned above, for film and television within Hollywood, the 3D capture technology that is currently used can be costly and hard to access by others. This is because the large, custom camera and lighting rigs are located in dedicated facilities leading to longer capture times and more costly use. With our products, more industries will have access to 3D facial scanning, with multiple deployments of our product for every city, due to the designed desktop form-factor of our products and usage of commodity devices. Our product also works for a wider range of use cases allowing for more capture.
Fostering Human Connections Between End-Users and Brands
80.lv: Do you think that your technology can foster human connections between end-users and brands? How can the tech you build facilitate brand awareness and brand loyalty?
Dr. Abhijeet Ghosh: With these 3D scans, there comes a sense of realness and further connection between what consumers are seeing on screen and reality. This technology will also help facilitate brand awareness and loyalty as facial scanning will provide companies with the ability to provide consumers with more accurate information, differentiating depending on the industry. For example, for skin care, depending on how a brand utilizes the technology, they’ll be able to take a consumer's facial scan and analyze what best suits their skin type. This will provide more accurate results, leaving consumers loyal to the brand. In-store scanning can help brands offer personalized recommendations to their clients and market new products.
Lumirithmic's Roadmap
80.lv: What's your current roadmap? What did you plan for 2022? What will we hear again from you?
Dr. Abhijeet Ghosh: Our current roadmap involves deploying our product for various industries: Gaming, Films, Ad-tech, Skincare, and Metaverse.
Every face is a target market for us. Our target is to scan millions of faces in the next 1-2 years using strategic partners and clients by adding value to their digital and/or real assets sales. We are developing strategic downstream integrations which will help millions of faces to enter the gaming, ad-tech, skincare, and metaverse worlds.