See the Augmented Reality Checkout Experience in Action

2X Winner at World’s Largest FinTech Hackathon

At the 2016 Money20/20 Hackathon, my team of 4 competed against 105 other teams and placed 1st in the competition. As Design Lead, I conceptualized a seamless retail experience starting from an Augmented Reality (AR) iPhone shopping app and ending with checkout through a physical point of sale (POS) terminal or digitally through the app. After creating pencil wireframes, I used Sketch to make user interface mockups for the iOS mobile app and ingenico POS terminal. I prepared assets for the app and used Unity to design the AR overlay. Finally, I edited a demo video and helped with the demo during the first round sponsor pitches and finalist presentations.

This was my second year winning at the Money20/20 Hackathon.

  • Role: Lead Designer
  • Timeline: 24 hours
  • Platform: iOS, Ingenico point-of-sales system
  • Tools: Sketch, Adobe Photoshop, Unity, Zeplin
  • Team members: 4

About the 2016 Money 20/20 Hackathon

The third annual Money20/20 Hackathon brought together over 650 developers, designers, and entrepreneurs to compete for $125K in cash, international fame, and additional sponsor prizes. MasterCard, Visa, WorldPay, ingenico Group, First Data, and Modo were some of the companies that participated this year.

To compete, participants created a new app in 24 hours, used APIs and other tools provided by leading financial companies, and presented their working creations to a panel of judges. Projects were scored on simplicity, creativity, impact, and design, all weighted equally.

Challenge

Create a new commerce experience for in-store, online, or mobile purchases, using the ingenico group SDK:

  • In-store: Build merchant or consumer-facing business apps that run directly on our new payment terminals using HTML5, CSS, and JavaScript.
  • Online: Build cross-border or domestic e-commerce checkout experiences for web or mobile.
  • Mobile: Build iOS or Android mobile apps to accept card-present payments with our Bluetooth-connected mPOS card readers.

Ideation

Inspired by the challenge, my team began envisioning a new checkout experience using augmented reality. Collaborating with the product owner, I explored the retailer’s business requirements and the consumer’s needs.

As a Retailer, I want to…

  • …offer real time promotions, so that I can push sales from real-time data based on seasonal trends, low sales volume items, and monthly/year end quotas.
  • …customize my style inspiration feeds, so that I can access new revenue streams through advertising and affiliate marketing channels.
  • …offer flexible payment options for my customers, so that they can seamlessly checkout from my store.

As a Customer, I want to…

  • …avoid long lines, so I can avoid the frustration of having to wait.
  • …have multiple payment options, so that I can choose how I want to pay.
  • …take part in a delightful shopping experience, so that I can enjoy how I spend my time.

I made sketches, annotations, and user flows in my notes below.

Wireframing

After planning with the team, I began sketching wireframes for a checkout experience encompassing the iPhone 7 and Ingenico Group’s Telium TETRA, a Point of Sales System.

Logo Design

My team and I deliberated on the team name, and eventually came upon hovershop, a fitting name for the experience. I sketched potential logomarks that expressed the product’s brand. I explored symbolism representing commerce, discovery, and location.

Mockups

After walking through wireframes with the team, I created high-fidelity designs for the checkout experience on both the mobile app and Telium TETRA point of sales system.

The checkout experience begins on a customer’s iPhone.

To checkout, a customer hovers over a product with his/her phone and taps on the circle indicator as it pulses. An augmented reality object pops up, displaying the product name, size, price, current promotions, social media styling inspiration, and an Add to Bag button. Once the customer adds the product to his/her bag, the total product quantity and total price are updated on the top right of the UI.

Once the customer taps the Checkout button on the mobile app, s/he is prompted to hold the phone over a POS device, in order to scan the QR code given by the POS device. (Alternately, the customer can also check out using the mobile app.)

Once scanned, the product data from the phone is transferred to the POS device, where the customer can finish the transaction.

The POS device displays an order summary with the subtotal, promotions, and taxes.

After reviewing the total price, the customer swipes his/her credit card to complete the transaction.

Upon successful payment, a final confirmation is shown with the total amount paid.

UI Implementation

Using Unity, I placed user interface (UI) components into the iPhone 7 frame. UI components included icons, buttons, boxes, and social media content.

Here’s a quick video of the Augmented Reality scanning implementation. See below for the full demo.

See the Augmented Reality Checkout Experience in Action