There are many types of testing that you can add at different phases of your project. Your last line of defense is testing within the final stages of the project—where you can catch those last-minute defects that can potentially harm your product’s use, damage its value to the user, and tarnish your company’s reputation as a manufacturer of high-quality items.
Various tools exist to assist you during these final stages. One of the most well-known tools for testing iOS apps is TestFlight.
If you’ve never used TestFlight, this post will be an introduction to the tool and will describe how you can best use it during your project to ensure few to no defects affect your final product.
What is TestFlight?
TestFlight is an online tool that allows you to install and test mobile apps “over-the-air.” Historically, app developers tested their projects on actual devices that had to be connected to the development machine using a wire to get a new build. Over-the-air allows developers to distribute their latest builds to testers without the need to connect to the phones physically. Because of this, new builds can effectively be sent to testers in any part of the world.
Who should use it?
TestFlight is perfect for small businesses that don’t have access to a large in-house testing team. And since it’s free, it allows for more businesses other than just revenue-generating companies to use it.
But it can also be good for enterprise companies with remote testers. The access and ease of distribution to up to 10,000 external users and the ability to gain their feedback is extremely useful to companies of all sizes.
When was TestFlight released?
On December 23, 2010, TestFlight was founded by Benjamin Satterfield and Trystan Kosmynka. At the time of its initial release, TestFlight was a single platform designed with the intention of testing mobile apps for both Android and iOS devices.
In 2012, TestFlight was acquired by Burstly, which raised a considerable amount of capital from venture capital firms in order to develop further features and launch TestFlight Live (a now discontinued service providing the user with real-time analytics and engagement metrics).
In February 2014, Apple acquired Burstly. By March of that year, they terminated support for Android devices.
How has TestFlight changed since being acquired?
Today, TestFlight is one of the many options for you to use to test your iOS app. Although there are alternative systems, a few of these still utilize TestFlight to do their app build distribution, especially since the limit for external testers was increased to 10,000 users. With that number of users, you’ll be sure to receive adequate feedback and get enough testing done to ensure a high-quality level of your app.
Currently, you can use TestFlight only for apps developed for Apple devices, as it’s available only to developers within the iOS Developer Program.
Once signed up for this service, you can distribute your app builds to internal or external beta testers who can provide you with feedback about the app. Along with this service, the TestFlight software development kit also allows developers to receive remote logs and crash reports from testers. Being able to gather data from users when they have experienced an issue can be invaluable, as some defects can be difficult to reproduce. The data from these crash reports could provide you with information to help you track down the issue and fix it before it affects anyone else.
At what stage should you use TestFlight?
Although TestFlight would be great during the earlier phases of development to test your initial builds, you need to upload a release build in order to use TestFlight. Using release builds to perform initial testing on isn’t typical, because they contain less debug information. In the event of a crash, you won’t have sufficient details in order to step through the code, find out what happened, and try to fix it.
Release builds are created when the developer is satisfied that the major bugs have been caught and it’s ready to be a beta candidate build. This is a close representation of the final version that will go to customers. Generally, the build is stable with few to no crashes. Any issues encountered at this phase usually require only minor tweaks, so release builds are better for beta candidate builds.
You can also use TestFlight after release to get live information about your app’s performance and feedback from users. Gaining ratings and reviews once an app is live is challenging, so giving users a way to easily submit feedback may help combat this issue.
What are the benefits and drawbacks of TestFlight?
While the history behind the product’s evolution gives you an idea of its features, there are many reasons why you may, or may not, want to choose TestFlight for your testing phase.
Benefits
Wide range of users
Being able to distribute your app to up to 10,000 external users over-the-air gives you the opportunity to get feedback for your app from a large number of users that you wouldn’t have previously been able to reach.
Having a beta testing group at this maximum amount is most common with game companies, as they have prospective users registering ahead of time to test the latest versions of a game iteration.
For normal apps, this number of testers is very large. I would typically expect a maximum of a couple hundred beta testers. Because remember, the more testers you have, the more test results you have to analyze and feedback you’ll need to review.
Keeping the number of beta testers to something sensible is recommended, no matter what maximum limit TestFlight provides.
Testing on multiple apps at once
You can test up to 100 apps at once with either internal or external users. You can also upload different iOS app builds—watchOS apps, tvOS apps, and iMessage apps—at the same time.
Internal users
Up to 25 members assigned to your team in iTunes Connect can also test your app on up to 30 devices.
Unlike external testers who will have access only to the build that their assigned link points to, internal testers will have access to all builds of your app. If you have multiple versions of your app—for example, a free and paid-for version—the testers could have access to each build of these different versions so you can get one person to test multiple apps.
Quick distribution
The tool allows you to quickly distribute your app to a specific set of beta testers over-the-air immediately. You may choose to build different versions to perform A/B testing of your app with your testers. Being able to send a specific build to a number of users at once will help you gain more results faster.
Easily accessible
TestFlight is integrated within the Apple Developer Dashboard that you need to use in order to distribute your app within the App Store. You only need to switch tabs to gain access; no separate login is required.
Drawbacks
iOS only
If you’re developing a cross-platform tool, this will allow you to test and receive feedback easily only on your iOS build. You’ll have to do additional work if you want to test the Android build.
Extra review needed to distribute via TestFlight
Apps that use TestFlight will require a Beta App Review and must comply with the full App Store Review Guidelines before testing can begin.
This is because your app is effectively going out to a select group of the public, so the Apple team wants to ensure that the build that is distributed is of the same quality as the current releases in the App Store.
Just like with the normal app submission process, if your app has significant changes between versions, you’re required to resubmit it for review. This will add extra time to the last phases of your app’s development before it reaches the public.
After you have completed beta testing, you will need to submit your app for review through the usual iTunes Connect screens.
Limited time for beta testing
Submitted builds are accessible from TestFlight for only 90 days after invitations to the testers have been sent. The app will stop writing after these 90 days, so you’ll have to either update your beta submission or get all of your testing completed within those 90 days. This restriction doesn’t affect your internal testers, only those external to the project.
Restricted to specific builds
Only release builds that are signed with the correct provisioning profile and distribution certificate are allowed to be uploaded for beta review. So the build needs to be signed appropriately in order for it to be built and accepted.
Only for devices running iOS 8 and above
A new iOS version is released every year, but not everyone upgrades and not everyone can upgrade, because Apple chooses to drop support for “old” versions of iOS hardware within about three years. This does slightly limit the reach to your audience, as you have no idea what devices they are using and whether their devices support iOS 8.
No application programming interface for continuous integration/continuous deployment
Is TestFlight for you?
Testing your app is a fundamental step in the process of developing one. Without testing, you’re blindly throwing your creation into the wild with no assurance that it will even work for your potential customers. TestFlight is a robust tool for iOS testing that’ll help you manage your beta testing stage effectively and get it out to potentially thousands of users.
I can completely understand the excitement to develop your idea into something tangible, but you must keep in mind that the goal is to create a great app, not simply to distribute anything that resembles your idea into the app stores.
To do this well, you need to remember that integrating thorough testing (even if you’re a team of one) into your development is essential for building a robust product and successful project. And gaining feedback from beta testers via TestFlight before your product is opened up to everyone could reveal some key insights.
This testing could change some big features and could possibly put your app on the course to becoming extraordinary. So while there are several drawbacks to using TestFlight, I think the positives ultimately outweigh the negatives and make it worth considering TestFlight as a resource for beta testing your next app.
Disclaimer: All views and opinions expressed in this post are those of the author based on his/her personal experience, and have not been endorsed, sponsored, or otherwise compensated by the companies or products mentioned therein.