29 Dec 2020

Unity, iOS, TeamCity, AppCenter

You can take the developer out of the tools company, but weaning the developer of tool making is another thing entirely. As per usual, at framebunker we remain focused on applying tools development where sensible to increase the efficiency multiplier on all efforts involved in development. This of-course includes a continuous integration setup, where we find another JetBrains solution to fitting our needs very nicely: TeamCity.

Aside from being able to rely on prior experience, we enjoy the flexibility afforded by the solution, suite of available plugins, and direct integration with other parts of our setup. The party trick? Integration in Rider allows not just running of build configurations based on what is already checked in to some branch, but also based on your current local state - for “hey, check this out!” and similar scenarios. Is this offered by other CI setups? Don’t know - probably? Is it cool? Very yes.

The [inaudible] X factor

While we find ourselves working in many different environments from time to time, we seem to gravitate towards Unity and iOS. When you add continuous integration to the mix, that means dealing with the Xcode cli for building and signing something actually useful in distribution. And this is where I wrote my TODO on writing this post. If you take a look around the wider interwebs for clues on how this interface works, you will come across a broad range of advice on how such a feat is achieved. All different, most fully correct at time of writing.

Historically things have changed in this area at least every couple of years which seems to pretty much coincide, with some padding, with how often I need to work with it. So every time I need to set something up, the environment has changed. Obviously underlying technology changes quite naturally over time, but in most cases that does not lead to people also radically changing the interface for said technology - desktop operating systems deal in windows, icons, and cursors, most cars have four wheels and a steering wheel, etc. - even if the technology underlying both has changed quite a bit over the years. Not so in this case. Exaggeration? Probably, but not by a whole lot.

Overview

Anyway, this is not the prime topic of the post - just part of its inspiration. I wanted to describe the whole flow outlined in the title of this post and provide some insight at each step. Hopefully you will find useful bits here and there even if the whole flow is not relevant to your needs - just as I picked up bits and pieces from other posts as well as the very helpful AppCenter support department (comparatively not a lot of community material out there, given how young the product is).

Overall the flow goes like this:

  • An iOS title is being developed in Unity. For us this means version 2019.4 - the latest LTS release at time of writing.
  • Collaboration happens via git - pushed to and pulled from a local server. Specifics are largely irrelevant to this flow, but in our case we use Gitea - including its support for git-lfs. Providing related support is an instance of the Unity cache server.
  • TeamCity is connected to the relevant project repositories and some build configurations are set to trigger as soon as fresh commits are detected. Among other things this provides us with up to date builds of the project available for download via the web interface.
  • For iOS builds that is not necessarily all that useful (aside from verifying that things build and pass tests) as you need to run an install flow in order to get on-device testing. Further, everyone relevant for distribution of test builds do not necessarily have access to our build server. So we rely on AppCenter (formerly Hockey App - at least this portion of it) to handle distribution of builds.

All of the above runs in Docker containers on our service network - with the exception of the macOS and Windows build agents which run in VMs on developer machines (enjoying that lovely trend of high core count desktop CPUs). In either case configurations easily transfered to other infrastructure - for example when dealing with client projects.

And why not do that? All of the above is definitely available for “free” or a minor fee, running on infrastructure maintained by someone else. Mostly flexibility. Running on someone else’s computer is great when you first start it up and everything lines up perfectly with your needs at the time. Reality, that is at least ours, unfortunately rarely remains on static spec - at which point maybe you can upgrade to a different plan or migrate to a different service, but more likely you end up just sucking it up and compromising your needs in order to not rock the service boat. Without spending time on depressing detail we have similarly had bad and potentially disastrous experiences with service operator support quality. Suffice to say I have no plans to again invite a session well past midnight of jumping VPN exit nodes to circumvent overzealous protection schemes while frantically laboring to salvage our data out of the fire of “computer says no”.

Unity to iOS

Building Unity projects on a CI setup means running the editor via command-line interface - loading the appropriate project in the right import target mode, cache server connected, invoking the appropriate editor script function you write to initiate the build you want, and finally routing output properly back to the CI system. Not a ton of work, but since a “runner” plugin for Unity projects already exists for TeamCity, we decided to not reinvent that particular wheel - relying on this plugin hopefully seeing some support. Our experience with it has unfortunately not been super smooth, so we may revisit that decision in the future.

A couple of things to keep in mind:

  • “Build and run” is what you would usually click in the Unity editor UI in order to end up with Xcode building a nice iOS app for you. Not so when going via Unity build pipeline API. In stead the flow is:
    • Build (not run) - this produces an Xcode project for you.
    • Optional modification of the generated Xcode project - utech has a library available for this.
    • Instruct Xcode to build the generated (and possibly further modified) Xcode project via command-line interface.
  • While the build function in your editor script must be static, the class implementing it cannot be. Similarly you should avoid nesting it further than one level in namespacing. Failing any of these will lead to Unity not being able to invoke the function via its command-line parameter.
  • Builds must take place inside one call stack originating in the function invoked via command-line - so being reliant on processes which need to be requested via Unity API before then being collected via an EditorApplication.update invocation or similar, issuing the final build call, is a no-go. This may well be a result of how the TeamCity plugin structures the actual command-line invocation of Unity, which is part of why we have a review of that plugin scheduled.
  • Over a number of Unity updates we have unfortunately seen a loss of some build-related functionality. Our solution has been to patch out these regressions directly in the UnityEditor assemblies, but that may not be an option for everyone. I hope to be sharing that setup and such patches in the not too distant future, but we’re not in a position where work on that front pays the bills - so no timeline promises. Build-relevant issues at time of writing (LTS 2019.4):
    • Preprocessing scenes is no longer possible - only postprocessing is.
      • Our patch bypasses the scene postprocess setup entirely and invokes our own, with the same sort of attribute markup, as appropriate. Internally it relies on a callback injected before Unity runs its scene processing - thus facilitating preprocessing. I am not aware of a non-patching alternative.
    • As promising as it was, the quite new compile pipeline API invokes its events before the produced assemblies are actually moved into place on disk. As a result, any further work on said assemblies is at best not possible (no assemblies there) or at worst defeated (assemblies are modified, only to immediately be overwritten with newly generated copies). Bypassing this issue is further complicated by IL2CPP which makes timing even more critical (no longer a matter of “late enough”, but also “before IL2CPP”).
      • Our patch relies on the compiler pipeline API as a source of “what has been built” (usually “everything”, so gains vs. just assuming “all of the assemblies” are minimal), but otherwise completely bypasses it. For playmode patching we just respond to domain unload and for builds a new callback is injected at a point just prior to IL2CPP invocation, when updated assemblies are known to have been written to disk. I am not aware of a non-patching alternative - particularly when IL2CPP is in play.
    • Aside from patching the Unity editor with these things (and other non-build-related stuff), we really enjoy the advantages of build-time changes to project assemblies afforded us by this post-processing flow. More than just code generation (like generating an enum based on Unity layer names), being able to modify, expand, and contextualise the actual code flow based the complete project - not just based on code structure, but actual usage as dictated by asset data - is quite exhilarating. Anyway, that’s an aside - not really build related. As stated - more on this in a later post (tm).

We implement the invocation of our Unity build script as one of two build steps (using the Unity plugin) of our TeamCity iOS build configuration. The other is the invocation of the Xcode CLI for building, signing, and packaging the Unity-generated project. This step is configured as the Command Line runner type, with the following “custom script”:

xcodebuild archive -project "output-path/Unity-iPhone.xcodeproj" -scheme Unity-iPhone -archivePath "output-path/name.xcarchive"

At time of writing that is really it! If you have worked with the Xcode CLI before or trawled the interwebs in search of this one line, you know how much other work has previously been involved. Fingers crossed that the above remains supported - if not static - for years to come. This of-course assumes that your build agent has already been configured to properly build and sign iOS applications when you invoke manually from the Unity editor UI.

The result of this step is an xcarchive - almost what we need for distribution.

iOS to AppCenter

Whether you need to iterate with on-device builds or are sharing progress with the team or external groups, you will eventually need to distribute your build. Signing configuration of-course has to be configured via Xcode and associated online services and I am sure it is possible to automate distribution that way as well - ultimately via TestFlight on the client side. However since that route is currently not mandated, we prefer to keep our multi-platform flow joined for as long as possible - hence AppCenter (or, originally HockeyApp - business will business).

Signing configuration aside, this is our only external service dependency. Given the subject matter, I am inclined to give this one a pass - especially as it means not needing to maintain an additional externally accessible network when distributing to external groups, and the much more manageable hit we would sustain should the service fail us. That said, I am certainly curious about self-hosted options.

While our iOS build configuration in TeamCity is set to trigger when new changes are detected, distribution is either a manual or scheduled step (“gimme NAO” vs nightly builds). Dependent on the project, several distribution configurations exist - one for each group “team”/”testers”/”this partner”/”that partner” as well as “me”. We rely on the TeamCity build configuration template and parameters features to avoid too much duplication between related, but specialised configurations. In the case of the AppCenter ones a template defines the parameters which it then uses in its “command line” build step, while inherited configurations define different values of those parameters (or use the default). For example, the “me” configuration defines the AppCenterGroup parameter as Direct-CI-%teamcity.build.triggeredBy.username%, which the template uses in the “custom script” of its “command line” runner:

COMMAND='appcenter'
COMMAND="$APPCENTER_CLI_INSTALL_PATH$COMMAND"

zip -ry Project.%env.BUILD_NUMBER%.zip Payload
mv Project.%env.BUILD_NUMBER%.zip Payload/Project.%env.BUILD_NUMBER%.ipa
echo "Starting upload now"
$COMMAND distribute release --file "%system.teamcity.build.workingDir%/Payload/Project.%env.BUILD_NUMBER%.ipa" --app %AppCenterApp% --group "%AppCenterGroup%" --token %AppCenterToken%

These configurations are based on the output of the earlier described iOS build configurations. As such they rely on a snapshot dependency from there, with the rule: Project.iOS.*.xcarchive.zip!Products/Applications/**=>Payload/. This instructs TeamCity to collect the zip of the previously generated xarchive and from it copy out the contents of the Products/Applications subfolder - into a local Payload folder. A couple of things to be aware of both in this and how it gets used in the above script:

  • The choice of the “Payload” folder name is convention. A valid ipa file is a zip archive with the application contained in a subfolder by this name.
  • Moving the zip file into the Payload folder after compression is not related to the upload process at all. In stead, I simply make use of the fact that I have the build configuration artifact dependency configured to “Clean destination paths before downloading artifacts”. So by moving the archive in there I make sure that the build agent clears it along with the uncompressed app when next it runs this configuration. Note that this does not conflict with having the build configuration use the ipa as its artifact.
  • Note that while the earlier script example was cross-platform (aside from Xcode being macOS-only), this is a shell script written for Linux or macOS. Initially I had this be a batch script run on our Windows build agent, but since we wanted the distribution step to always be available (as opposed to whenever a developer machine with a Windows build agent was online) and the Windows environment ironically wasn’t ideal for running this tool, I moved it to an Ubuntu-based Docker [container][teamcity-appcenter-agent-docker which could run along with TeamCity and our other services.
  • While zip is the program you want for zip compression under Ubuntu, on the Windows 10 command-line, you need to use tar - which conversely does not produce zip archives on Ubuntu.
  • appcenter-cli does not provide any status output - hence the echo command. A --debug flag is available, but turning it on only outputs the REST data sent and received - no additional error information or status reporting results.
  • Not all AppCenter tokens are created equal. Make sure to use appcenter-cli to create the token eventually supplied to its distribute invocation - tokens created via the web UI are not valid for this use.

Build agent setup

As mentioned, while the TeamCity server itself runs in a Docker instance, the build agents performing the brunt of the actual build-related tasks generally run as VMs on developer machines - with the above exception. This gives us much more free hands to configure the build environments for the tasks needing performed - aside from setting up and running the build agent software itself.

For the AppCenter build agent that means installing nodejs and the appcenter-cli tool. A couple of notes on that:

  • While the appcenter-cli documentation does mention that it requires nodejs 10, that is not a minimum spec, but an absolute. Installing latest nodejs and hoping for the best will, at time of writing, only lead to long debugging sessions.
  • When setting up a Windows-based build agent for AppCenter use, you will want to not just install the appcenter-cli node package as a global package, but first modify the nodejs “prefix” from the default location somewhere in the current user path to something global (I go for C:\nodejs) - ensuring build agent accessibility regardless of its user mode now or later.
  • You may have noticed the APPCENTER_CLI_INSTALL_PATH variable used in the distribution script. This is just an environment variable set on the build agent - not something specific to TeamCity like the configuration parameters. Not only does this facilitate easy adaptation to different build agents offering the same capability - it is also usable as an agent requirement defined in the TeamCity build configuration. APPCENTER_CLI_INSTALL_PATH points to where appcenter-cli is installed, so any agent with that environment variable set should be able to run build configurations dependent on that tool.

For other agents (or the same - if you do not want to run appcenter-cli on unique agents), we just need a licensed Unity install of the appropriate version, the TeamCity agent software, and a Java runtime to run it. macOS agents with iOS-build capability will also need Xcode and its cli installed. Notes:

  • The TeamCity install provides the agent software - just navigating to the agents overview in the web UI provides an install link. Once the agent is up and running (having been authorized by an administrator via the web UI), the TeamCity install takes care of keeping the agent software up to date.
  • While Windows agents get a nice little UI installer, the macOS ones are a bit more manual. Minimal steps to get things going at time of writing are as follows (just pasted from our wiki):
    1. Unzip buildAgent.zip file to ~/buildAgent.
    2. Configure ~/buildAgent/conf/buildAgent.properties:
      • serverUrl=http\://[DNS-or-IP]\:[PORT]
      • name=[MachineName]-macOS(optionally function and number)
      • workDir=[full user folder path]/buildAgent/work
      • tempDir=[full user folder path]/buildAgent/temp
      • systemDir=[full user folder path]/buildAgent/system
      • env.TEAMCITY_JRE=[full user folder path]/buildAgent/jre
      • ownPort=9090
    3. Create the folder ~/buildAgent/logs
    4. Load the agent for initial connect and auto-upgrade from the server sh buildAgent/bin/mac.launchd.sh load
    5. (Administrator authenticates the agent via the web UI)
    6. Copy ~/buildAgent/bin/jetbrains.teamcity.BuildAgent.plist to ~/Library/LaunchAgents/
  • Further / more up to date details are available in this section of the TeamCity documentation.
  • The days of Java being easily available on macOS are unfortunately gone. Disregard the java install flow suggested by macOS / Xcode and in stead install / fix Homebrew, using it to install the necessary Java version (at time of writing Java 8):
brew cask install adoptopenjdk8
  • If opting to run appcenter-cli in a QUEMU/KVM based VM, do note that its upload process seems especially susceptible to problems in the default VM driver for the NIC. Making sure that this is set as virtio in stead eliminated some hefty delays when we were still running in a full VM-based agent.

Wrap and future

In general we are pretty happy with this setup. Iteration speed is really snappy, it serves all our needs, and the full flexibility facilitates some pretty cool options. Services are backed up off-premises on the same schedule as everything else and support delays & limitations are zero. Obviously establishing things took a bit more doing than signing up for some services and connecting them, but that cost is nowhere near a place where I would consider trading in all the upsides we have found.

Aside from that review of the Unity TeamCity plugin, we are not likely to be doing a ton of tinkering with this part of our service setup in the immediate future, but one opportunity I did come across while setting up the AppCenter flow was managing distribution group composition via the cli. If so inclined, it might be interesting to have an infrequently run configuration use this to synchronise distribution groups with the Xcode signing groups - especially in a scenario where large groups of recipients are managed and frequently modified. Not currently relevant for us, but it seemed like an interesting project.

Anyway, that’s about it. I hope you found something useful in this overview and best of luck in whatever it is you do.

New Gear
Unity, iOS, TeamCity, AppCenter
RAID0 NVMe on Ubuntu
A Change of Gears
CoreObject
Unity Protocol Buffers
Behave 2.7
Learn
Behave 2.6
Trusted Gear
Mad Mash Versioning
Behave 2.5
Behave 2.4
Co-very-routine
Construct
The Engine Wars: Numbers
GDC 14: The Quest For Fun
Moving in Unity
Behave 2.3
Unity and .net assemblies
Behave 2.2
ReView
Behave 2.1
Behave 2.0
Unity Hacks: Dual sticks
Unity Hacks: Cameras
Unity Hacks: Touch gestures
OnRenderTextureGUI
Unite 13 video "Unity Hacks" available
The implicit local network interface
Talks and progress
Five years of Unity expertise looking for contracts
Automagic Unity Android Java gadget OF DOOM!
Invading Planet from your couch
Mountain Lion and laggy bluetooth and duct-tape
Unite 12 video and new videos section available
Asia Bootcamp videos now available
Path is now MIT licensed
Behave 1.4 released
So I've been a bit busy lately
Behave 1.3 released
IGDA Unity SIG slides
Second Unity IGDA SIG this evening: Scene construction and AI
First IGDA Unity SIG this evening
Alternative licensing available
Pathfinding in two lines
Path 2 released
Assembling and assimilating
Path 2 intro screencast
Path 2 beta release for GGJ
AIgameDev master class video now online
Expanding beta
Behave AIgameDev master class public stream
Behave master class on open AIgameDev stream tomorrow
Interview with AIGameDev
New video: From tree to code
Issue tracking on github Behave release project
IT University Copenhagen Unity course completed
IT University Copenhagen Unity course files Thursday
CPH IT University Unity course files
Behave 1.2 released
Video: Behave - starting from scratch
Behave runtime documentation updated
Behave 1.1 released
FAFF cleanup: Sketch
Building a menu of delegates and enums
Pick me! Pick me!
Optimising coroutine yielding in C#
Downloading the hydra
New license of Path: GPL
CopyInspector
Magnetic
GUI drag-drop
Logging an entire GameObject
I bet you can't type an A!
Where did that component go?
UnitySteer
New and improved: Behave 1.0 released
Behave 0.3b and unity 2.5
Behave 0.3b hotfix
Path tutorial video available
Path 1.0 launched!
Continued community tutorials
Community tutorial
New tutorial
First tutorial available
Behave 0.3b
unite '08 open-mic session
Behave 0.2b
Behave 0.1b
Behave pre-release
Path beta 0.3b
Path beta 0.2b
Path beta 0.1b
Path pre-release