Solving the CTV User-Agent Problem

Solving CTV User-Agent String Problem

While OTT consumption is increasing, so too is CTV device fragmentation. This has led to many challenges and opportunities. But before we jump into these, we need to limit confusion around the terms OTT and CTV. So first let’s look at the recently published formalized distinction between OTT and CTV.

The terms OTT (Over The Top) and CTV (Connected TV) are unfortunately used interchangeably. This leads to confusion and difficulties when finer details are being discussed. As the OTT and CTV ecosystem has grown over the years so has the need for distinction between the terms.

To enhance clarity within the OTT and CTV AdTech ecosystem, Amit Shetty, Senior Director, Product at the IAB Tech Lab, has recently provided a formal distinction between the two terms.

Difference between OTT and CTV

OTT vs CTV | CTV UA String Problem Image

Source: iabtechlab.com

 - OTT

OTT refers to content. The term OTT should be used when the device type is irrelevant. As can be seen in the image above, OTT Streaming Services, such as Netflix and Hulu, can be viewed on any device including mobile and desktop.

OTT continues to distinguish premium television content from online video.

 - CTV

CTV refers to a category of device types. This category includes streaming devices that are connected to a TV and TVs that are capable of streaming. As can be seen in the image above, Smart TVs and Game Consoles are CTV device types, while phones, tablets, laptops and desktops are not.

This formal distinction between OTT and CTV will help reduce confusion in the OTT and CTV AdTech ecosystem, when fully adopted. The IAB, having developed the distinction, are leading the way in its adoption. Notably, the “OTT Technical Working Group” has been renamed and is now more aptly called “CTV Technical Working Group”.

Ok, so now that we’re on the same page regarding these terms, let’s take a look at the trends in OTT consumption and CTV devices, along with key challenges, solutions and opportunities within the OTT ecosystem.

Trends In OTT Consumption And CTV Devices

OTT consumption continues to increase. eMarketer estimates that, in the US, the average time spent consuming subscription OTT video content in 2020 will surpass 1 hour per day. That’s a 23% increase from 2019. This is due to many factors including the speed up of cord-cutting, stay-at-home orders and the growth and entry of new OTT Streaming Service providers. Media giants such as Disney, HBO Max and NBC have brought with them massive content libraries, which is helping to drive the increase in consumption of subscription OTT.

Average time spent consuming subscription OTT content | CTV UA String Problem Image | Source: eMarketer

Source: eMarketer

The above graphic, from eMarketer, shows and estimates the average time spent consuming subscription OTT video in the US, from 2018 to 2022.

This growth is echoed by ComScore research, which also provides insight into CTV devices used for consumption. Displayed in the graphic below, ComScore’s fourth annual State of OTT webinar notes that reach among traditional devices (e.g. phone, desktop, tablet) is declining, albeit slightly, while CTV is increasing.

CTV Device Reach Compared Traditional Devices | CTV UA String Problem | Source: ComScore

Source: ComScore – State of OTT 2020

Interestingly, “Streaming Box/Sticks” is now on par with “Tablet”, followed closely behind by Smart TV. ComScore continued and noted that, with a 22% growth, Smart TVs are the fastest growing streaming segment of OTT content:

CTV Device Growth | Streaming Devices vs Smart TVs vs Game Consoles vs Other OTT Devices | Source: Comscore

Source: ComScore – State of OTT 2020

The 6% drop in Gaming Consoles, shown above, is likely due to the increase in time spent playing video games during Covid-19, and therefore less time available / being spent streaming OTT content. Statista notes that “during a March 2020 survey, video gamers in the United States reported that they spent 45 percent more time playing video games amid the quarantine than in the previous week.”

Other key takeaways from ComScore’s State of OTT webinar highlight:

  • “Drops in household reach for live TV and DVR, but increases in OTT and VOD”
  • “Total time spent on OTT has just begun to exceed total time spent on DVR”
  • “This broadcast season, hours per household spent with OTT are higher than hours per household on DVR”

To learn more, access ComScore’s State of OTT 2020 webinar and report, here.

The OTT ecosystem is relatively new and has developed rapidly. This, combined with the continued growth in OTT consumption and the increasingly fragmented CTV device landscape, has led to many challenges.

A key issue that the CTV Technical Working Group are now tackling relates to User-Agent strings. DeviceAtlas were more than happy to share our expertise with IAB to help tackle this major problem.

Below we take a look at this problem and the issues that it generates, along with DeviceAtlas’ contribution and the solution.

The CTV User-Agent String Problem

IAB’s CTV Technical Working Group members, according to the IAB, have expressed that “a large number of CTV impressions are incorrectly reported as not CTV”. This is primarily due to issues with User-Agent strings. The negative result includes:

  • Issues with targeting, triggering and rendering ad creative, analytics and reporting;
  • Reduced buyer and publisher confidence; and
  • Negative impact on monetization.

To help overcome the User-Agent string problem in the OTT ecosystem, we were delighted to have the opportunity to contribute input to the IAB.

Unique Position and Expertise

Here at Afilias Technologies Limited, a wholly owned subsidiary of Afilias PLC and developers of the DeviceAtlas and DeviceAssure products, we deliver market leading solutions to industry and specialist insight to industry standards.

Our partnerships and involvement with leading industry bodies, including the IAB, ITU and GSMA, along with our contributions to open web standards (e.g. editors of 3 open standards with W3C, and  IETF contributors) and open source projects (learn about our Android 10 TAC contribution here) enables us to help ensure that industry and end users of connected devices leverage best-in-class solutions and standards.

In this regard, we were delighted to share our expertise to help solve (and experience in solving) the CTV User-Agent string problem. We have long been providing recommendations to customers and partners in many ecosystems about optimal User-Agent string formation. Additionally, DeviceAtlas already partners with a range of top players in the OTT ecosystem, such as SpotX, AppNexus/Xandr and Netflix, to overcome User-Agent string challenges & leverage device intelligence opportunities.

 - Expertise

In order to deliver accurate device intelligence, our team of data specialists analyse and process millions of User-Agent strings every month – over 10,000,000 unique User-Agent strings last month alone (i.e. May 2020). Through our network of sites (in excess of 200,000 websites), data partners and customer submissions, we ensure that we capture all new User-Agent strings and index them against device, along with its capabilities. On average, our team of specialists add 1,000 new devices per month.

Ultimately, our unique position meant that we were well suited to help tackle the CTV User-Agent string problem.

Solving The CTV User-Agent String Problem

With the aim overcoming the User-Agent string problem and the negative effects noted above, the IAB’s CTV Technical Working Group were seeking to standardize the User-Agent string format.

 - Our Contribution In Brief

As mentioned, we were delighted to contribute to the Working Group. Some of our insights shared included key CTV User-Agent issues, which are listed briefly as follows.

Insights Into Common CTV User-Agent String Issues

  1. Absence of model identifiers

The model identifier is the key piece of information required for effective device intelligence. The below examples are from non-browser apps installed on Apple and Android devices (note: actual app names were removed from the User-Agent strings and replaced with “Appname#” tokens). What they have in common is that there is very little usable information in the User-Agent strings. 

Appname1/14 CFNetwork/978.0.7 Darwin/18.7.0

Appname2%20GO/538 CFNetwork/978.0.7 Darwin/18.7.0

Appname3/3.1.0-leanback (Linux;Android 7.1.2) Appname3Lib/1.5.11

Appname4/1909.1 Android/7.1.1 crocus
  1. Including irrelevant information

The inclusion of irrelevant information / tokens in User-Agent strings is common and against IEFT guidelines.

  1. Encoding issues

Example strings:

  1. Spaces replaced by ‘+’ symbol
Mozilla/5.0+(Linux;+Android+5.1.1;+AFTT+Build/LVY48F;+wv)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Version/4.0+Chrome/70.0.3538.110+Mobile+Safari/537.36

Dalvik/2.1.0+(Linux;+U;+Android+5.1.1;+AFTT+Build/LVY48F)+(Philo+TV+Fire/3.3.9-amazon) 
  1. Characters encoded
Mozilla%2F5.0+%28iPad%3B+CPU+OS+12_4_1+like+Mac+OS+X%29+AppleWebKit%2F605.1.15+%28KHTML%2C+like+Gecko%29+Mobile%2F15E148

String encoding usually happens after its creation, when, for example, the User-Agent string is passed within a url from one service to another. However, some services / platforms seem to encode the strings on an early stage and pass it further in an encoded form.

Encoding may change strings in various ways, therefore, it affects the string parsing by either adding a negative performance impact or by hampering correct string identification. As a result, monetisation is impacted.

  1. Model collisions

When the model number is included, but with no identification of brand, the risk of collisions occurs where different brands utilise the same model numbers. For example, there are more than a dozen brands with a model “M3”. Inclusion of identification of brand in the User-Agent string resolves this.

CTV Device Image

In addition to sharing our insights from User-Agent string issues that we’ve encountered and overcome, a key input that we contributed was our recommended User-Agent string format. This formed the foundation of discussion for formalizing (for preliminary guidelines) the User-Agent string. After valuable input from CTV Technical Working Group members, we provided a modified recommendation. The end result being as follows.

 - Recommended CTV User-Agent String Format

In order to overcome the issues mentioned above, it is recommended (as part of the IAB Tech Lab’s OTT/CTV User Agent Preliminary Guidelines) that the following User-Agent string format is used.

<device info> <os name>/<os version> <app name>/<app version> <other info>

For example:

DeviceBrand DeviceModel OSName/1.2.3 AppName/1.2.3 LibName/1.2.3

 - Summary Of The Benefits Of Using Well-formed User-Agent Strings

  1. Improved monetisation

Since the device forms an important part of targeting, particularly with the rise in privacy concerns globally reducing access to user-level information, a well-formed User-Agent string increases the value of impressions since more precisely targeted campaigns can execute.

  1. Correct representation in analytics platforms

If a device / OS / browser cannot be identified via the User-Agent string, it will not be correctly represented in web analytics platforms. As a result, the device / OS / browser will not be factored into marketing or business decisions by organisations utilising analytics for their business planning.

  1. Optimized user experience

Leading content publishers use both client side and server side tools to optimize the user experience, so that engagement with their content is maximised. In the event that the device / OS / browser cannot be identified on the server side, a key tool is unavailable to the publisher and the end user may receive a suboptimal experience, for example in the form of oversized or generic content.

 - IAB’s OTT/CTV User Agent String Preliminary Guidelines

It was a pleasure to work with the CTV Technical Working Group to help formalize a User-Agent string format for the IAB Tech Lab’s OTT/CTV User Agent Preliminary Guidelines. And it is great to see, from our User-Agent string processing, that some companies have already implemented the formalized format.

We believe that, if fully explored and adopted, the guidelines will result in a significant step forward for the industry and its end users. We’re excited to see the positive impact, and we encourage all relevant parties to consider the adoption of the guidelines and to add to this conversation. Download the OTT/CTV User Agent Preliminary Guidelines, here.

And to help with the conversation, we also encourage the adoption of the distinction between OTT and CTV noted above. Learn more about the distinction in Amit Shetty’s “OTT vs. CTV: What’s In A Name?” article, here.

CTV Device Intelligence Solution

DeviceAtlas supports every step in the chain, from publishers (Netflix, CBS Interactive), through monetization (Xandr, SpotX, The Trade Desk) to analytics (Adobe, Conviva). We help a range of top players overcome User-Agent string challenges and leverage the opportunities enabled by real-time device intelligence.

Through our quick and easy to implement, high-performance API, DeviceAtlas enables AdTech companies to:

  • Accurately identify OTT traffic (e.g. Set Top Boxes, Smart TVs, Games Consoles);
  • Stay on top of the CTV device landscape more efficiently; and
  • Enhance inventory value by providing another detailed layer of targeting for advertisers.

Learn more about our market leading device intelligence solution & get a free trial, here.