A week prior to Christmas (2017), I was invited to Amazon’s Seattle headquarters for a first-hand demo of the Amazon Go concept along with an opportunity to speak with Dilip Kumar. Dilip is Vice President of Technology at Amazon, with responsibility for both Amazon Books and Amazon Go stores. Prior to taking over in his current capacity, he was VP, Technical Advisor to Jeff Bezos. The visit lasted about an hour, the first part dedicated to the store tour and demonstration and the last half for question and answers. During that time, I participated in an interactive demonstration of the store as well as a follow-on discussion that answered some of the questions I had prepared and those that arose during the demonstration. Since our meeting was around lunchtime, the store was quite busy and “Amazonians” moved rapidly in and out of the store. Since the store was not open to the public access was limited to Amazon employees. Located at the mothership, those coming in and out of the store seemed to be regular users. While there was an employee positioned at the door to address any issues and assist employees with loading the application, none of those visiting the store needed any help.

Besides the visit being relatively short, the other minor disappointment was that I was unable to take any photos. This was primarily due to the fact that the stores which will be rolled out to the public will have a somewhat different layout from the development store and they did not want to misrepresent the concept to the public. While most readers are probably familiar with the published video introducing the concept, which came out about a year ago, I’ve linked it here as a refresher. The picture below is a screen capture of the entry/identification process. The thing that can easily be overlooked is the picture is the array of “boxes,” mounted just above the lights from the drop ceiling. We’ll get back to these.

For the demo, I was given an Android phone with the application loaded. While the tour was under NDA, we were given permission to speak about it today since the Amazon Go application should now be available for both iOS and Android and the store opens to the public on Monday, January 22. The Go application is linked to the user’s Amazon account which is tied to his or her credit card. The user opens the application, scans a QR code to enter the store, and you’re done. At that point, one can even turn their phone off and the items selected will be added to your virtual cart.

I have not explored the application, so I can’t provide detail on it. From a functional standpoint though, a natural question emerges with regards to the need to return items and how that would be handled. Returns are pretty simple. One can return something right from the application. He or she will begin on the honor system, but by using analytics, they will monitor this over time for issues. Their view is the vast majority of customers are honest and, for the small number who intentionally steal, they’ll be prosecuted in the same way any traditional store would. They will certainly have copious quantities of camera evidence.

As we walked around the store, I was given the option of selecting any items that I wanted. The product assortment was mostly ready to eat items, drinks, prepackaged snacks, and some condiment-type items. The food looked very appetizing, especially some of the locally sourced desserts. Unscientifically, I would estimate that the product mix was maybe 30% Go private label items like ready to eat sandwiches and snacks prepared there in the store (the prep area can be seen from the street as shown in the picture below), 10-15% locally sourced items, and the remainder were national brands. They did offer some Whole Foods brand specific items, but since I’m not a Whole Foods shopper (there is not one in my immediate geographic area), I could not estimate the percentage. Dilip was kind enough to note these Whole Foods items were part of the product assortment, which I found to be interesting since the Whole Foods acquisition wasn’t completed that long ago. I think more than anything, this speaks to the speed in which Amazon operates.

They do note vegan and gluten-free on the shelf tags, though these notifications do not currently synchronize with the Go application. That exclusion is done purposefully so that the application is non-intrusive.

There was also the option to purchase a small assortment of alcohol. Since it was lunchtime there were no takers there, but if I had been there at the end of the day, I assume it would have been a much more popular drink choice. There was a small corner section that was ‘roped’ off in a sense. It had just enough of a barrier to create a path where any partakers could interact with the Go employee who was checking identification. Some of the things I found to be pretty creative, and that I think will be immensely popular, are their chef-created meal kits. These meal kits come with all ingredients in one box and can be prepared in about 30 minutes. The selections change frequently based upon options created by an employee chef. The kits run in the $16-20 price point and feed two people. Ultimately, what they have done is give customers a good reason to come into the store multiple times a day by keeping the assortment different enough and bringing a TJX/Marshalls/Ross treasure seeking like component to the food shopping experience with the quick prep meals.

For the technology aspect of things, it was quite elegant as observed. My perspective is that the technology is essentially transparent and unobtrusive. Based upon my understanding of what is in the marketplace, what I observed, and Dilip’s comments, it appeared that they engineered all of the technology themselves. For those familiar with Amazon and their ways of doing business, this was not particularly surprising. Dilip indicated that their requirements were such that the hardware and software they needed was currently unavailable in the marketplace, which necessitated creating it themselves. Two major technologies they were forthcoming about were weight sensors on some of the fresh item shelves, and their camera technology. They have designed and built their own camera systems.

An inference was made that there might have been something else from a sensor perspective, but I was not able to observe anything additional. There was no use of RFID and none of the prepackaged items had any special tags or markings on them. As time progresses maybe we’ll learn more about this.

Their cameras were a little different. They were mounted on a drop ceiling, at a height of about 10-11 feet. For a visual, please refer to the red arrow in the second picture showing the consumer scanning his phone as he walks through the turnstile. The cameras appear as gray boxes and are on average placed about every 1 to 1.5 feet apart. Most of the cameras are mounted to view at an angle and a few of them point straight down. It would require more time to look at the products on the shelves relative to those cameras to understand why, but I think it is because some of the packaging is non-descript for computer vision. On some of the packages they have their own version of a barcode made up of small squares and triangles in a square grid. The cameras were about 1.5” x 7” x 9” with slits for what I would assume are cooling fans in the back.

Amazon designed the cameras and had them purposefully built to their own specifications. I have no misgivings that their main purpose was to house the optics of a camera lens, but I think they also run the custom developed image recognition software, yes, edge computing. This allows them to run the image analysis right there, at the point of detection, and then pass over to AWS where all of the summation/correlation activity is done. I’m guessing the camera placement may be a mixture of art and science, so with the potential for every store to have a slightly different layout, this placement may require some tweaking. I say this because cameras were not placed in a repeatable pattern, which to me would suggest some tuning will be necessary. Given that information, it is also not clear how different product assortments or packaging changes will impact camera placement.

When I asked about any limitations for the number of people in the store, I was told it is only limited by the fire code, not the technology. I would say there were on average 25-40 people most of the time over the 20 minutes I was in the store. This is an important point because there was speculation at this time last year that they were limited to 20 simultaneous people and thus the reason for the delay in roll out. If that was a problem, it clearly is no longer.

With regards to other limitations, I would say potentially there is a cost limitation. The cameras/CPU’s are mounted every foot and a half of linear shelf space, which probably costs $400-600 each to manufacturer. For another retailer considering the use of such technology, this would probably be cost prohibitive. I can’t imagine the cost to a full-sized grocery store would be reasonable and for that reason, Dilip said they have no plans to bring this technology/camera system into Whole Foods.

On the positive side of the high cost of checkout, you get almost perfect inventory visibility and a great way to reduce shrinkage. So, the technology itself does save money in some areas which helps offset the increased cost of the high density of the sensors.

One of the most interesting aspects to me was trying to get a feel for the accuracy. Again, from articles published over a year ago, the concerns I noted centered around read accuracy (that being the correlation of the items selected and what is reflected in your virtual cart) and simultaneous customer counts. Dilip would not give specifics, but he seemed to intimate that the real limitation was not customers in the store, but read accuracy. With regards to current read accuracy, he would not comment, but only would say that it was “very, very good.” I tried to ask this question in several different ways, but that was the best I could discern. My inference is that they have made pretty drastic improvements over the past year. I also infer that their accuracy is currently at an economically feasible level and their concept and technology is at a place where they can move the concept to the general public.

With regards to the roll out, the WSJ, among others, did note there was a recent job posting for an Amazon Go construction manager. When I asked about specific numbers, dates, locations, etc. Dilip declined to answer. He did laugh at the suggestion of 1,000 stores. That didn’t come as a surprise to me. The feeling I got was that this technology is going to be focused in major metropolitan areas and probably going to be a pretty modest roll out especially initially. I suspect we’ll hear much more about the details in the next several months, but the key thing to note is that the roll out will be much more reserved than initial estimates. Those stores will probably vary based upon available size of the space and one may or may not be able to see the food prep areas. Visibility of the food prep area was a very nice feature as anyone who is a Subway fan will tell you.

One of the more curious aspects will be future plans for licensing the technology. Dilip responded that currently there were no plans to do so, but it will be interesting to see how this plays out, and if they warm to the idea. This is important because as of now AWS is a major presence in the cloud, impacting a broad swath of retail. Potential licensing of the technology stack could pave the way for them to influence technology in the store. It will also be interesting to see if any of the major store technology providers or systems integrators reach out with interest in being a licensing partner.

As a technologist, what I saw was absolutely fascinating. I can see this technology will be something that will appeal to a very wide range of consumers. As mentioned earlier, one of the major undefined details is quantifying the accuracy. Like much of what Amazon does, it is their next step in raising the bar with consumers, and trying to envision for consumers what the purchasing process can be, either online or in store. While other technology companies have some nice pieces, from what I observed, Amazon was able to pull it all together, and in a pretty slick fashion. While there are definite limitations of this particular approach in apparel (how would computer vision be able to detect the difference between the same color and style shirt with one a M and the other a L) and would be too costly to scale to grocery or warehouse clubs (also an issue with ceiling heights and mounting proximity requirements), but with alternative sensor technology coupled with customer imaging and tracking in the store, I can definitely see where some of these concepts could be extrapolated to other verticals. As an analyst, it will be fascinating to watch, and I’m sure for most of our readers as well.