At the launch of the iPhone 11 series, Apple Senior Vice President Philip W. Schiller introduced the concept of computational photography for the first time when he introduced the iPhone 11 Pro series imaging system, and the concept became known to the public for the first time.
In fact, the concept of computational photography is not new. It first appeared in a public paper in 1994, and it was determined that in-camera composite HDR, panoramic photos, and simulated bokeh all belonged to the category of computational photography. The concept of computational photography is not new.
Philip W. Schiller, who introduced computational photography at the iPhone 11 Pro launch event.
A few decades later, the carrier of image recording changed from film to digital, cell phones had cameras, and computational photography emerged from theory to become a major trend.
But this trend has little to do with cameras, camera manufacturers are still on track to improve the pixel, continuous shooting speed and video capabilities, seemingly indifferent to computational photography, the resulting photos (straight out) are still very mediocre, gradually “surpassed” by smart phones.
On the contrary, the computing power of smartphone chips is getting stronger and stronger, AI, algorithms and machine learning are intervening in a wider range, and there are more and more ways to interpret images, and eventually the photos processed by a series of “algorithms” are getting better and better.
Now, many people are more willing to use cell phones to record and share, and cameras are becoming less and less common, which is also reflected in the market performance of the two, the smartphone market is growing strongly, while the camera market is shrinking year after year, and even DC (card cameras) are gradually disappearing.
At this point, some people will ask, since the smartphone with a shot from the photo so good, why the traditional camera manufacturers do not follow the trend of computing photography, consider improving the appearance of photos straight out?
Is the camera computing power is not enough to calculate over?
Let’s start with the “core” of this problem.
The core of the phone is the SoC, which integrates the CPU, GPU, ISP, NPU and baseband, allowing you to make calls, take photos, watch videos, play games, surf the Internet and other operations, which also directly determines the performance of the phone.
The core component of the camera is the image sensor (CMOS), which is similar to the phone except for the component area, for imaging and light sensing. In addition, the central processing chip that controls the entire camera system is called the image processor (Image Processor).
Sony’s BIONZ X image processor (α7 series royalty), for example, includes the SoC and ISP chip, and does not integrate the ISP in the SoC, the advantage is that Sony can increase the number of ISP chips on its own according to the performance needs of the CMOS (α7RIII BIONZ X is equipped with dual ISP), the disadvantage is that the degree of integration is not as high as cell phones.
The role of the SoC in the BIONZ X is similar to that of a cell phone, controlling the control interface and camera functions, but not requiring high performance. Bayer transformation, demosaicing, noise reduction, sharpening and other operations on the “data” collected by the image sensor are mostly dependent on the ISP, which eventually converts the data collected by the CMOS into real-time framing of the camera. In this process, the camera’s ISP is not involved in the calculation process, but only treats the photos as products on the assembly line and carries out uniform processing.
▲ Sony BIONZ X image processor.
With the number of pixels, continuous shooting speed, and video performance of cameras now increasing, the camera’s image processor has a high demand for speed and throughput of image processing, and the amount of single data is huge. Without involving “computation”, the processing power of the camera’s image processor far exceeds the processing power of the current smartphone ISP.
But when it comes to computational photography, or AI capabilities, it’s a little less so. The imaging process of a smartphone is somewhat similar to a camera, but before presenting the final image, it also requires ISP and DSP calculations, real-time adjustments and optimization, especially after the multi-camera system has become mainstream, the amount of computing data on the phone has grown exponentially.
After the iPhone 11 Pro series launched multi-camera system, multi-camera system can be smooth and seamless switching behind the A13 Bionic in the addition of two machine learning gas pedal huge data processing capacity, reaching a trillion times per second, so high-frequency and efficient data processing capacity is considered to eat the huge amount of data generated by the three cameras.
The image processor of the camera is mostly pre-processing of the original data, almost no calculation process, while the cell phone SoC includes data acquisition pre-processing and subsequent calculation process, the two focus on different directions.
For different groups, the result of market segmentation
The root cause of the rapid development of cell phone computational photography is still due to the small size of the image sensor (CMOS) of cell phones, with the current technology, want to physically exceed or close to the camera can only be optimized through algorithms, spell straight out of the view, for example, automatic HDR, super night scene, simulation of large aperture, magic for the sky and other functions.
The “calculation” process done by iPhone to take a photo.
The first thing you can do is to use the same algorithm that you use for your own personalized applications, such as filters, HDR highlights and darkness. However, for cell phones for the masses, as far as possible to let most people take good photos, more in line with the market positioning of cell phones and crowd positioning.
Since the invention of the camera, the camera has absolute “tool” attributes, in order to be efficient, the appearance, control, functionality and so on will be compromised to efficiency. The company’s main goal is to provide the best possible service to its customers.
The RAW files record more information and allow for a greater range of adjustments.
For most people without a basic knowledge of photography, it is far more important to have a good looking photo at hand than to have an informative photo. And for camera makers in the professional world, improving the color depth of RAW recordings is more in line with market positioning than improving JPG direct output.
However, things are not so absolute, and cameras are trying to change. Fuji has been working on the camera’s straight-out effect, introducing “film simulation” through different algorithms to make the photos more flavorful and better looking. However, this process is not calculated by the scene, but by the user’s own choice, which is somewhat similar to some film simulation apps for cell phones, and does not involve the so-called “computational photography”.
AI post-processing is the main direction of the camera?
In the field of photography, post-processing is an essential step. On the one hand, post-processing software can make full use of the rich information recorded in the RAW format, and on the other hand, it can also leverage the high performance and computing power of PCs to quickly process photos.
The first of these is the “RAW” format.
The newest version of Luminar 4 supports AI automatic sky change.
Adobe’s Photoshop has been updated in recent versions to include automatic recognition in operations such as keying, repairing and skinning, making the operation more and more brainless and the effect more and more accurate. The Pixelmator Pro retouching software on the Mac platform started to identify images with the help of Apple’s Core ML machine learning as early as 2018 to make color adjustments, keying, selection, and even use the ML machine learning engine when compressing the output.
▲ Pixelmator Pro 2.0’s image editing support is powered by the machine learning engine.
As mentioned earlier, camera manufacturers are hardly making an effort in computational photography due to chip AI algorithm limitations and niche market issues. But the explosion of AI in post-production software has also made up for the shortcomings of cameras in computational photography.
Even including the AI of post-production software, cameras still have not gotten rid of the traditional process, cameras record, software processing, the process is still cumbersome for the public. The company’s main goal is to provide a better solution to the problem.
The number of global digital camera shipments in September 2020 is far less than in 2018.
According to CIPA, the camera market is shrinking, while in contrast, the cell phone market continues to grow. The trend of “computational photography” on smartphones will not change the direction of cameras becoming more professional, nor will it reverse the shrinking of the camera market.
In other words, even if the cameras now have the ability to “computational photography” close to smartphones, can they save the “declining” camera market? The answer is, of course, no. To take an extreme example, if it is feasible to fight straight out, the Fuji camera will have the first market share. In fact, the first seat of mirrorless camera is now occupied by Sony, which is not a good-looking straight out.
The Sony microphone has become the working machine of many studios.
The company’s main goal is to provide the best possible service to its customers.
Increasing camera specialization also means the need for better-performing image sensors (CMOS), but “computational photography” relies on separate machine learning modules, and it is well known that chip development is costly and risky, making it difficult for camera manufacturers to balance the two. Computational photography and development specialization are two different paths, while “computational photography”, “AI intervention” and other features of little use to professional users are likely to be strategically abandoned by camera manufacturers for the time being to balance R&D costs.
At this stage or in the foreseeable future, camera manufacturers want to embrace “computational photography” high risk, large investment, slow results, is more difficult, not to mention that there are now a number of professional post-production software with AI retouching to support the bottom.
Recent Comments