New Gear

2021-12-30 00:00:00 +0100

In 2015 I did a review of my active toolbox. As obviously a lot has changed since then, particularly with pandemic-mode operations, I figured an update would be appropriate.

I had hoped to have done a post on a new build agent setup we are working on, but unfortunately JetBrains support are still having some trouble replicating problems we are experiencing with Linux-based TeamCity Unity build agents. You will just have to look forward to that - hopefully not too far into the new year.

A central theme in the current toolbox is self-hosting and flexibility in general. Third party provided services are great until you get near the edge of intended use cases - which usually works well, but holds the capacity to (hopefully only) slightly frustrate every project flow you go through. We first dipped our toes in self-hosting some services pre-pandemic but since moving to be fully distributed we went all in - to the point of currently only relying on external services for text chat, email, and mobile build distribution.

That said, I will strive to keep the format of the earlier post - focusing primarily on local machine tools. So as mentioned the update here is flexibility and obviously a use case change as well, given the steep drop in need for working on the go.

Platform

The 2013 MacBook Pro got retired - initially in favour of a company workstation at the office and a personal one at home, for once pieced together from parts matching my needs rather than the sales considerations of a manufacturer of prebuilts. Moving our operations fully distributed, resulting in the company workstation moving in with me, the personal workstation then got converted into a server - replacing the basic Synology box which had previously been sufficient for the limited services we self-hosted when sitting multiple non-family-members in offices were a thing.

This change of hardware platform obviously meant a move away from macOS - marking the first change in the toolbox. Prior to retiring the laptop I had started dual booting Ubuntu on it in order to gauge how far the Linux desktop experience had moved since last I had a look at it. A clear goal in general was an increase in flexibility - not just limited to hardware choice, so if given an actually modern and fully functional desktop experience I was definitely going to prefer Linux over Windows. At the end of my trial run I was satisfied with the experience and had gradually moved to cross-platform software on the macOS installation as well - in order to reduce the number of new things I would need to deal with when jumping completely.

After landing well on the new platform and properly acclimating I did move around a bit before settling on my current setup, but as a migration point Ubuntu (with the default GNOME desktop environment) was excellent. For anyone doing the research today I would recommend trying out Zorin OS as the migration point in stead - regardless of where you consider migrating from.

I prefer to not use headspace for more tools than necessary, so I opted to use Ubuntu server for our self-hosted efforts - in order to work with the same underlying tooling available in the terminal application of my desktop. The before-mentioned moving around did eventually see me leave Ubuntu for both use cases - in favour of NixOS as the underlying system, with Plasma as the desktop environment on my workstation. Note that while Plasma may initially look to be “the Windows experience but on top of Linux”, the primary selling point of the environment is accessible customisation. If I were to describe my current setup in more familiar terms, it would probably be: a macOS-like header bar & launcher, no dock, window auto-tiling, and a functionality-focused high-contrast if dark colour scheme. You can basically adjust it to your hearts content, making it an optimal fit for your particular flow - which is what drew me to it.

I would not recommend NixOS as a migration entry point, but long term the paradigm shift is well worth it to me. The advantages are numerous, but personally I am mostly drawn by the declarative OS configuration enabling a single text file to describe a complete system and all of its settings. The flexibility on Linux is great, but just like any other system there are a ton of different locations to review & configure the full experience (Windows settings applications meme: Engage!). Having a single file I can use to quickly review what a system is and to modify it is absolutely excellent. It is no animated semi-transparent GUI of smooth loveliness, but I’ll take it.

For control over hardware resources, service segregation, and advanced tools management I have opted to run a hypervisor on both server and workstation. Aside from massive cloud servers and game consoles, hypervisors are great for configuring and running a number of personal virtual machines on the same hardware - in particular when you have clearly segmented services and goals to work with. From a security standpoint network segregation in this environment is not as solid as with individual machines, given that virtual machine escape exploits are a thing. However neither my private nor our company threat model includes threat actors with the necessary budget and given that the alternative is third party services susceptible to the very same attacks, this configuration is entirely acceptable. This setup means that my day to day desktop experience is actually a virtual machine. Though with a dedicated graphics card and plenty of resources & CPU priority associated that distinction is entirely unnoticeable. In addition to this primary VM, the workstation runs a router for the different VMs, CI agents for the different projects, and other tools like distributed bake & generative services, devkit tools with picky environment requirements, and that sort of thing.

As an aside, I also ended up running NixOS on my tablet - though here I opted for GNOME rather than Plasma as the desktop experience. While explicitly not designed for touch devices, I find it the better experience of the two on such hardware. In addition to buttery smooth regular reading of books & comics + browsing + media activities, the gaming experience on the tablet is great - using Steam in-home streaming to run games from a VM on the server running an attached dedicated graphics card, with controls either via touch or the excellent Steam Controller. This on a minimum-spec dual-core device with four gigabytes of memory.

Tracking

OmniGroup only develop their software for the fruit company ecosystem, which meant I would of-course have to find an alternative to OmniFocus. Given the difficulty in finding a suitable project & TODO tracking solution described in the last toolbox post, this was by far the toughest workflow change.

Thankfully our self-hosting journey overlapped nicely with the local toolbox change here as I found an excellent combination of user experience and flexibility in the YouTrack solution we were already using for company project tracking. While a fully hosted version is available as well, we had already opted for self-hosted in order to more easily run some custom integration with the rest of our flow. The resulting setup is one install for the company, one for the family, and optionally one per project if it involves tightly integrated external parties.

The underlying data structures can have a whole lot of fields associated with each task/entry - making it a full Jira-like complex beast if that is what you want - or a simple TODO app if that is more your target. Combined with a mobile app and a very flexible system of searches/queries bookmarked or exposed in lists, dashboards, reports, and agile boards, I have found that I can easily tailor an experience to anything from a complex project through small efforts to grocery shopping without slamming into too many walls or facing highly “enterprisey” experiences for very simple use cases.

For flow integration we use the official .net API - as we as much as possible strive to standardise all our code- & data base to C# and markdown respectively. The experience here has been great as well. We have primarily used the API to update other systems based on changes in tasks, but our wishlist does currently include more bidirectional integration which should be fully supported by the API.

My calendar is still a central part of my day to day flow - essentially feeding tasks from the tracker through a planning phase and into the calendar. Continuing to use the iCloud calendar setup either through the web app or integrated with a local client is an option, but for forward-looking flexibility and a more stable experience I have opted for a self-hosted solution here as well. A couple of options are available here but probably of most broad appeal would be the CalDav service built into Nextcloud. So long as the service runs https, you can add the account to iOS just fine and on the desktop I have worked with a couple of different options until finally settling on the very lovely, but still-pre-release Kalendar project.

Typing

Since the last toolbox update the life of this typist has improved significantly. While Sublime Text is still great and a very important part of my toolbox, OmniSharp unfortunately did not last as a stable experience. Thankfully Rider picked up that slack and then some, offering a rock solid IDE with an extensive suite of tools and of-course neat integration with the other JetBrains products in our arsenal. As a tool for .net and Unity development, Rider has been a no-brainer for a while. Praise aside, particularly for our IL patching workflows I am not really satisfied with the assembly/IL browsing functionality of Rider when it comes to external assemblies rather than the assembled result of a loaded project. For this need I am still a very happy camper with the Avalonia ILSpy project.

Worth noting when speaking of Sublime Text is Sublime Merge - another newcomer since last update. While Fork is still excellent and would be my first recommendation to anyone, Sublime Merge offers better platform support and an interface striking just the right balance for me between simplification and utility.

Given that we standardise on markdown as much as possible, expanding the toolbox with a dedicated markdown editor only made sense. While it offers a lot more than just that, Obsidian is my pick for markdown editing. At time of writing the WYSIWYG editor is very freshly released, but none the less it is my daily driver for the purpose.

Searching

Developed specifically for the fruit company ecosystem, Dash is another previously mentioned tool in need of replacement. Thankfully the developer is very open to clients on other platforms having great access to the largely-community-supplied documentation packages. The consequence is a healthy selection of alternatives, of which I went for Zeal. Available on Linux, macOS, and Windows, this offline documentation viewer is just as excellent a tool to me as Dash was on macOS.

I honestly did not really have much use of the editor integration of Dash in the past, so only having the global shortcut + search tags for accessing Zeal was not a real change for me. Integration may be available, but given the base I was operating on I just did not go looking for it. Thankfully more alternatives are available and perhaps even a fully editor embedded client - in case that is more relevant to you.

Before the platform & services shift mentioned throughout this write-up, I decided to complete my workstation ergonomics by looking at peripherals which for some inexplicable reason I had ignored through years of paying attention to desk, lighting, seating, and monitors. Perhaps the reason is that this aspect of ergonomics has an odd air of fringe vs. the other aspects.

Previously I had been using a flat wireless keyboard with short key travel and a nicely large wireless touchpad. There are a ton of options and semi-religious opinions in this area, but after a while I settled on a vertical wireless mouse (a regular mouse where your hand orientation ends up more vertical than regular horizontal) and a mechanical split keyboard. I found the ErgoDox open source design very compelling and the number of available options based on it equally so. Aside from being split, the layout is also column stagger rather than the default row stagger - resulting in a more natural primary vertical movement of your fingers rather than constant sideways switching.

Once I had the board in hand I realised the further room for ergonomic optimisation. Since the available boards implementing the ErgoDox design are fully customise-able via the QMK project, you can use visual editors to redefine key functions as you like or even dive fully into the C-based firmware and write your own custom layout & -functionality there. I used this opportunity to further reduce finger stretch by removing use of some keys entirely and moving other functionality to layers (while not accurate, if you think if the shift key as changing the keyboard layout to a layer of capital letters and different symbols, you have the general idea) - including a numpad and pre-baked shortcut keys like the copy key you double-tap to cut or the word left/right keys positioned just above the left/right arrow keys - which again are flanked by the line start/end keys.

Combining these ergonomic layout optimisations with baked shortcut keys and function keys mapped in Plasma to different functionality in different applications (like the macOS shortcut mapping demonstrated in the last toolbox post), my work environment took a significantly step forward on ergonomics, functionality, and sheer accessible speed.

As a bonus, the ErgoDox board I opted for came with per-key LED lighting. As I understand it you are supposed to utilise this for fancy animated effects, providing a clear speed & precision boost when gaming - and while I could certainly use both, I opted instead for using it to highlight key functions in the active layer - like a clear inverted T-shape for the arrow keys or a grid for the numpad etc.

Wrap

And that is about it. All these years later I do not have any current conference plans, but my toolbox certainly keeps adapting to my current needs. Overall I feel much more efficient and flexible in my day to day and am happy to have a feeling of being able to do much more with my local and extended environment - to adapt to personal and company needs on any given effort. I hope that this update has helped inspire you - if nothing else then to review your current toolbox - perhaps to see if anything in there could stand a revisit compared to available alternatives today.

Unity, iOS, TeamCity, AppCenter

2020-12-29 00:00:00 +0100

You can take the developer out of the tools company, but weaning the developer of tool making is another thing entirely. As per usual, at framebunker we remain focused on applying tools development where sensible to increase the efficiency multiplier on all efforts involved in development. This of-course includes a continuous integration setup, where we find another JetBrains solution to fitting our needs very nicely: TeamCity.

Aside from being able to rely on prior experience, we enjoy the flexibility afforded by the solution, suite of available plugins, and direct integration with other parts of our setup. The party trick? Integration in Rider allows not just running of build configurations based on what is already checked in to some branch, but also based on your current local state - for “hey, check this out!” and similar scenarios. Is this offered by other CI setups? Don’t know - probably? Is it cool? Very yes.

The [inaudible] X factor

While we find ourselves working in many different environments from time to time, we seem to gravitate towards Unity and iOS. When you add continuous integration to the mix, that means dealing with the Xcode cli for building and signing something actually useful in distribution. And this is where I wrote my TODO on writing this post. If you take a look around the wider interwebs for clues on how this interface works, you will come across a broad range of advice on how such a feat is achieved. All different, most fully correct at time of writing.

Historically things have changed in this area at least every couple of years which seems to pretty much coincide, with some padding, with how often I need to work with it. So every time I need to set something up, the environment has changed. Obviously underlying technology changes quite naturally over time, but in most cases that does not lead to people also radically changing the interface for said technology - desktop operating systems deal in windows, icons, and cursors, most cars have four wheels and a steering wheel, etc. - even if the technology underlying both has changed quite a bit over the years. Not so in this case. Exaggeration? Probably, but not by a whole lot.

Overview

Anyway, this is not the prime topic of the post - just part of its inspiration. I wanted to describe the whole flow outlined in the title of this post and provide some insight at each step. Hopefully you will find useful bits here and there even if the whole flow is not relevant to your needs - just as I picked up bits and pieces from other posts as well as the very helpful AppCenter support department (comparatively not a lot of community material out there, given how young the product is).

Overall the flow goes like this:

An iOS title is being developed in Unity. For us this means version 2019.4 - the latest LTS release at time of writing.
Collaboration happens via git - pushed to and pulled from a local server. Specifics are largely irrelevant to this flow, but in our case we use Gitea - including its support for git-lfs. Providing related support is an instance of the Unity cache server.
TeamCity is connected to the relevant project repositories and some build configurations are set to trigger as soon as fresh commits are detected. Among other things this provides us with up to date builds of the project available for download via the web interface.
For iOS builds that is not necessarily all that useful (aside from verifying that things build and pass tests) as you need to run an install flow in order to get on-device testing. Further, everyone relevant for distribution of test builds do not necessarily have access to our build server. So we rely on AppCenter (formerly Hockey App - at least this portion of it) to handle distribution of builds.

All of the above runs in Docker containers on our service network - with the exception of the macOS and Windows build agents which run in VMs on developer machines (enjoying that lovely trend of high core count desktop CPUs). In either case configurations easily transfered to other infrastructure - for example when dealing with client projects.

And why not do that? All of the above is definitely available for “free” or a minor fee, running on infrastructure maintained by someone else. Mostly flexibility. Running on someone else’s computer is great when you first start it up and everything lines up perfectly with your needs at the time. Reality, that is at least ours, unfortunately rarely remains on static spec - at which point maybe you can upgrade to a different plan or migrate to a different service, but more likely you end up just sucking it up and compromising your needs in order to not rock the service boat. Without spending time on depressing detail we have similarly had bad and potentially disastrous experiences with service operator support quality. Suffice to say I have no plans to again invite a session well past midnight of jumping VPN exit nodes to circumvent overzealous protection schemes while frantically laboring to salvage our data out of the fire of “computer says no”.

Unity to iOS

Building Unity projects on a CI setup means running the editor via command-line interface - loading the appropriate project in the right import target mode, cache server connected, invoking the appropriate editor script function you write to initiate the build you want, and finally routing output properly back to the CI system. Not a ton of work, but since a “runner” plugin for Unity projects already exists for TeamCity, we decided to not reinvent that particular wheel - relying on this plugin hopefully seeing some support. Our experience with it has unfortunately not been super smooth, so we may revisit that decision in the future.

A couple of things to keep in mind:

“Build and run” is what you would usually click in the Unity editor UI in order to end up with Xcode building a nice iOS app for you. Not so when going via Unity build pipeline API. In stead the flow is:
- Build (not run) - this produces an Xcode project for you.
- Optional modification of the generated Xcode project - utech has a library available for this.
- Instruct Xcode to build the generated (and possibly further modified) Xcode project via command-line interface.
While the build function in your editor script must be static, the class implementing it cannot be. Similarly you should avoid nesting it further than one level in namespacing. Failing any of these will lead to Unity not being able to invoke the function via its command-line parameter.
Builds must take place inside one call stack originating in the function invoked via command-line - so being reliant on processes which need to be requested via Unity API before then being collected via an EditorApplication.update invocation or similar, issuing the final build call, is a no-go. This may well be a result of how the TeamCity plugin structures the actual command-line invocation of Unity, which is part of why we have a review of that plugin scheduled.
Over a number of Unity updates we have unfortunately seen a loss of some build-related functionality. Our solution has been to patch out these regressions directly in the UnityEditor assemblies, but that may not be an option for everyone. I hope to be sharing that setup and such patches in the not too distant future, but we’re not in a position where work on that front pays the bills - so no timeline promises. Build-relevant issues at time of writing (LTS 2019.4):
- Preprocessing scenes is no longer possible - only postprocessing is.
  - Our patch bypasses the scene postprocess setup entirely and invokes our own, with the same sort of attribute markup, as appropriate. Internally it relies on a callback injected before Unity runs its scene processing - thus facilitating preprocessing. I am not aware of a non-patching alternative.
- As promising as it was, the quite new compile pipeline API invokes its events before the produced assemblies are actually moved into place on disk. As a result, any further work on said assemblies is at best not possible (no assemblies there) or at worst defeated (assemblies are modified, only to immediately be overwritten with newly generated copies). Bypassing this issue is further complicated by IL2CPP which makes timing even more critical (no longer a matter of “late enough”, but also “before IL2CPP”).
  - Our patch relies on the compiler pipeline API as a source of “what has been built” (usually “everything”, so gains vs. just assuming “all of the assemblies” are minimal), but otherwise completely bypasses it. For playmode patching we just respond to domain unload and for builds a new callback is injected at a point just prior to IL2CPP invocation, when updated assemblies are known to have been written to disk. I am not aware of a non-patching alternative - particularly when IL2CPP is in play.
- Aside from patching the Unity editor with these things (and other non-build-related stuff), we really enjoy the advantages of build-time changes to project assemblies afforded us by this post-processing flow. More than just code generation (like generating an enum based on Unity layer names), being able to modify, expand, and contextualise the actual code flow based the complete project - not just based on code structure, but actual usage as dictated by asset data - is quite exhilarating. Anyway, that’s an aside - not really build related. As stated - more on this in a later post (tm).

We implement the invocation of our Unity build script as one of two build steps (using the Unity plugin) of our TeamCity iOS build configuration. The other is the invocation of the Xcode CLI for building, signing, and packaging the Unity-generated project. This step is configured as the Command Line runner type, with the following “custom script”:

xcodebuild archive -project "output-path/Unity-iPhone.xcodeproj" -scheme Unity-iPhone -archivePath "output-path/name.xcarchive"

At time of writing that is really it! If you have worked with the Xcode CLI before or trawled the interwebs in search of this one line, you know how much other work has previously been involved. Fingers crossed that the above remains supported - if not static - for years to come. This of-course assumes that your build agent has already been configured to properly build and sign iOS applications when you invoke manually from the Unity editor UI.

The result of this step is an xcarchive - almost what we need for distribution.

iOS to AppCenter

Whether you need to iterate with on-device builds or are sharing progress with the team or external groups, you will eventually need to distribute your build. Signing configuration of-course has to be configured via Xcode and associated online services and I am sure it is possible to automate distribution that way as well - ultimately via TestFlight on the client side. However since that route is currently not mandated, we prefer to keep our multi-platform flow joined for as long as possible - hence AppCenter (or, originally HockeyApp - business will business).

Signing configuration aside, this is our only external service dependency. Given the subject matter, I am inclined to give this one a pass - especially as it means not needing to maintain an additional externally accessible network when distributing to external groups, and the much more manageable hit we would sustain should the service fail us. That said, I am certainly curious about self-hosted options.

While our iOS build configuration in TeamCity is set to trigger when new changes are detected, distribution is either a manual or scheduled step (“gimme NAO” vs nightly builds). Dependent on the project, several distribution configurations exist - one for each group “team”/”testers”/”this partner”/”that partner” as well as “me”. We rely on the TeamCity build configuration template and parameters features to avoid too much duplication between related, but specialised configurations. In the case of the AppCenter ones a template defines the parameters which it then uses in its “command line” build step, while inherited configurations define different values of those parameters (or use the default). For example, the “me” configuration defines the AppCenterGroup parameter as Direct-CI-%teamcity.build.triggeredBy.username%, which the template uses in the “custom script” of its “command line” runner:

COMMAND='appcenter'
COMMAND="$APPCENTER_CLI_INSTALL_PATH$COMMAND"

zip -ry Project.%env.BUILD_NUMBER%.zip Payload
mv Project.%env.BUILD_NUMBER%.zip Payload/Project.%env.BUILD_NUMBER%.ipa
echo "Starting upload now"
$COMMAND distribute release --file "%system.teamcity.build.workingDir%/Payload/Project.%env.BUILD_NUMBER%.ipa" --app %AppCenterApp% --group "%AppCenterGroup%" --token %AppCenterToken%

These configurations are based on the output of the earlier described iOS build configurations. As such they rely on a snapshot dependency from there, with the rule: Project.iOS.*.xcarchive.zip!Products/Applications/**=>Payload/. This instructs TeamCity to collect the zip of the previously generated xarchive and from it copy out the contents of the Products/Applications subfolder - into a local Payload folder. A couple of things to be aware of both in this and how it gets used in the above script:

The choice of the “Payload” folder name is convention. A valid ipa file is a zip archive with the application contained in a subfolder by this name.
Moving the zip file into the Payload folder after compression is not related to the upload process at all. In stead, I simply make use of the fact that I have the build configuration artifact dependency configured to “Clean destination paths before downloading artifacts”. So by moving the archive in there I make sure that the build agent clears it along with the uncompressed app when next it runs this configuration. Note that this does not conflict with having the build configuration use the ipa as its artifact.
Note that while the earlier script example was cross-platform (aside from Xcode being macOS-only), this is a shell script written for Linux or macOS. Initially I had this be a batch script run on our Windows build agent, but since we wanted the distribution step to always be available (as opposed to whenever a developer machine with a Windows build agent was online) and the Windows environment ironically wasn’t ideal for running this tool, I moved it to an Ubuntu-based Docker [container][teamcity-appcenter-agent-docker which could run along with TeamCity and our other services.
While zip is the program you want for zip compression under Ubuntu, on the Windows 10 command-line, you need to use tar - which conversely does not produce zip archives on Ubuntu.
appcenter-cli does not provide any status output - hence the echo command. A --debug flag is available, but turning it on only outputs the REST data sent and received - no additional error information or status reporting results.
Not all AppCenter tokens are created equal. Make sure to use appcenter-cli to create the token eventually supplied to its distribute invocation - tokens created via the web UI are not valid for this use.

Build agent setup

As mentioned, while the TeamCity server itself runs in a Docker instance, the build agents performing the brunt of the actual build-related tasks generally run as VMs on developer machines - with the above exception. This gives us much more free hands to configure the build environments for the tasks needing performed - aside from setting up and running the build agent software itself.

For the AppCenter build agent that means installing nodejs and the appcenter-cli tool. A couple of notes on that:

While the appcenter-cli documentation does mention that it requires nodejs 10, that is not a minimum spec, but an absolute. Installing latest nodejs and hoping for the best will, at time of writing, only lead to long debugging sessions.
When setting up a Windows-based build agent for AppCenter use, you will want to not just install the appcenter-cli node package as a global package, but first modify the nodejs “prefix” from the default location somewhere in the current user path to something global (I go for C:\nodejs) - ensuring build agent accessibility regardless of its user mode now or later.
You may have noticed the APPCENTER_CLI_INSTALL_PATH variable used in the distribution script. This is just an environment variable set on the build agent - not something specific to TeamCity like the configuration parameters. Not only does this facilitate easy adaptation to different build agents offering the same capability - it is also usable as an agent requirement defined in the TeamCity build configuration. APPCENTER_CLI_INSTALL_PATH points to where appcenter-cli is installed, so any agent with that environment variable set should be able to run build configurations dependent on that tool.

For other agents (or the same - if you do not want to run appcenter-cli on unique agents), we just need a licensed Unity install of the appropriate version, the TeamCity agent software, and a Java runtime to run it. macOS agents with iOS-build capability will also need Xcode and its cli installed. Notes:

The TeamCity install provides the agent software - just navigating to the agents overview in the web UI provides an install link. Once the agent is up and running (having been authorized by an administrator via the web UI), the TeamCity install takes care of keeping the agent software up to date.
While Windows agents get a nice little UI installer, the macOS ones are a bit more manual. Minimal steps to get things going at time of writing are as follows (just pasted from our wiki):
1. Unzip buildAgent.zip file to ~/buildAgent.
2. Configure ~/buildAgent/conf/buildAgent.properties:
  - serverUrl=http\://[DNS-or-IP]\:[PORT]
  - name=[MachineName]-macOS(optionally function and number)
  - workDir=[full user folder path]/buildAgent/work
  - tempDir=[full user folder path]/buildAgent/temp
  - systemDir=[full user folder path]/buildAgent/system
  - env.TEAMCITY_JRE=[full user folder path]/buildAgent/jre
  - ownPort=9090
3. Create the folder ~/buildAgent/logs
4. Load the agent for initial connect and auto-upgrade from the server sh buildAgent/bin/mac.launchd.sh load
5. (Administrator authenticates the agent via the web UI)
6. Copy ~/buildAgent/bin/jetbrains.teamcity.BuildAgent.plist to ~/Library/LaunchAgents/
Further / more up to date details are available in this section of the TeamCity documentation.
The days of Java being easily available on macOS are unfortunately gone. Disregard the java install flow suggested by macOS / Xcode and in stead install / fix Homebrew, using it to install the necessary Java version (at time of writing Java 8):

brew cask install adoptopenjdk8

If opting to run appcenter-cli in a QUEMU/KVM based VM, do note that its upload process seems especially susceptible to problems in the default VM driver for the NIC. Making sure that this is set as virtio in stead eliminated some hefty delays when we were still running in a full VM-based agent.

Wrap and future

In general we are pretty happy with this setup. Iteration speed is really snappy, it serves all our needs, and the full flexibility facilitates some pretty cool options. Services are backed up off-premises on the same schedule as everything else and support delays & limitations are zero. Obviously establishing things took a bit more doing than signing up for some services and connecting them, but that cost is nowhere near a place where I would consider trading in all the upsides we have found.

Aside from that review of the Unity TeamCity plugin, we are not likely to be doing a ton of tinkering with this part of our service setup in the immediate future, but one opportunity I did come across while setting up the AppCenter flow was managing distribution group composition via the cli. If so inclined, it might be interesting to have an infrequently run configuration use this to synchronise distribution groups with the Xcode signing groups - especially in a scenario where large groups of recipients are managed and frequently modified. Not currently relevant for us, but it seemed like an interesting project.

Anyway, that’s about it. I hope you found something useful in this overview and best of luck in whatever it is you do.

RAID0 NVMe on Ubuntu

2019-12-07 00:00:00 +0100

A bit of an odd topic after a period of silence, but it’s something that annoyed me greatly recently and so I wanted to spare others that experience. Meanwhile, what has been keeping the updates? framebunker and StaticSky have! In case you missed it, I started (slowly) brushing off and open-sourcing some of our work there (yes, I’ll do what I can to move through my “to open source” list faster): https://github.com/framebunker/Utilities

Update January 2020: Found & updated the link to the blog post which mentioned the appropriate sectors to clear on SSDs for md. Also one of the people who assisted in the Keybase team, David Hauken, wrote a post on this topic.

Challenge

So anyway, I set up a new computer at home - the age of my 2013 laptop showing. On that I wanted to run Ubuntu off a full-disk encrypted drive. However a core point of the build is speed and full-disk encryption (via LUKS) does not necessarily go hand in hand with that.

To offset that, I decided to go for a configuration with two PCIe4 NVMe sticks running in RAID0 (well cooled). Simple enough in principle, but then you hit the community coverage on the topic of “RAID0 is dangerous! Remember to back up your things and don’t pour water on your hardware”. It would appear that an Ubuntu (or event Linux) configuration like this is not super common.

Luckily there were enough bits and pieces out there to, in combination with invaluable assistance from the people of the sysadmins_unite Keybase team, string together a solution.

Process

If, like me, you are migrating an existing install to this setup then first make a backup of that install. I just used rsync - making sure that all file flags etc. were left intact in the copy. In creating the RAID0 across the two NVMe sticks any data already present on either will of-course be lost.

Next, burn the Ubuntu installer onto a bootable medium. This approach probably work on a ton of other configurations, but given that I have only used it with Ubuntu 18.04, I am going to describe them in that known-to-work context.

Boot into your installer and fire up a terminal. In these steps it is assumed that the two NVMe sticks mount at /dev/nvme0n1 and /dev/nvme1n1. If that is not the case in your setup, replace in the correct name where used. On my setup I was able to verify the mount points both in this installer terminal and in the Asus BIOS.

The first step is to install the mdadm tool - for creation and management of md RAID configurations. Note that critical to this particular usecase is the clearing of superblocks and overwriting of first one gigabyte of data. Most guides I came across would instruct clearing data at the end of the target drives, but one (~~I unfortunately lost the link - sorry about that~~ never mind - found it in an old window of research tabs) pointed me to the first gigabyte. As I understand it, this is specific to how md operates on SSD or at least NVMe.

Configuring your RAID:

sudo apt-get install mdadm
sudo mdadm --zero-superblock /dev/nvme0n1
sudo mdadm --zero-superblock /dev/nvme1n1
sudo dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=1000
sudo dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=1000
sudo mdadm --create --verbose --level=0 --raid-devices=2 /dev/md0 /dev/nvme0n1 /dev/nvme1n1

With your raid up and running in your installer session, I really hope you did not forget to copy data you really wanted off those sticks. Anyway, the next step is to run the Ubuntu installer, opting to do “something else” when typical vs. not-quite-typical install configuration is chosen. This takes you to a screen where you get to specify which drives and partitions are get used for what in your fancy new install. Necessary config:

Format your new /dev/md0 drive. For me that was formatting to LUKS and then adding an ext4 partition to that. You could also format directly to ext4 if you do not want the encryption, but in that case you might want to do that outside of the installer - for optimum performance. See the later section for specifics on that.
Specify that the installer use your final ext4 partition as root /.
Use a different drive and partition for EFI and /boot. This is critical as your bios will not know how to assemble the md RAID and boot from it - you need /boot to handle that. There were some very creative suggestions in sysadmins_unite on where one might put /boot, but I simply have a SATA drive in my rig used for more rough work and first tier backup. I just set up /boot and EFI partitions there.

That’s it - configure the rest of the installer as you like and run it to completion. When it does complete and prompt you to restart, don’t do that. The RAID still only really exists within this installer session so if you reboot now, you might as well start over.

The Ubuntu installer will have your new install partition mounted at /target. You need to make some edits to it in order to have it properly mount and run off your RAID from boot. If you closed the terminal from before, open up a new one and run this:

sudo mount /dev/[partition used for /boot] /target/boot
sudo mount --bind /dev /target/dev
sudo mount --bind /sys /target/sys
sudo mount --bind /proc /target/proc
sudo cp /etc/resolv.conf /target/etc/
sudo gedit /target/etc/default/grub

In gedit you need to update the value of GRUB_CMDLINE_LINUX to "domdadm", save, and close the editor.

Finally you need to add mdadm to your new install, which will also apply your grub change along with its own to the boot setup, after which we are good to reboot and test the install:

sudo chroot /target
apt-get install mdadm
exit
sudo reboot now

If you just wanted a fresh install, then this is the “profit!” step. Otherwise, restore your previous install onto your new root partition and update it to support your new boot setup by rebooting into the installer again and running this in a terminal:

sudo apt-get install mdadm
sudo mdadm -A -s
sudo mkdir /target
sudo mount /dev/md0 /target
sudo mount --bind /dev /target/dev
sudo mount --bind /sys /target/sys
sudo mount --bind /proc /target/proc
sudo cp /etc/resolv.conf /target/etc/
sudo chroot /target
apt-get install mdadm
exit
sudo reboot now

And that is your “profit!” step.

ext4

If you just want to run ext4 as fast as possible directly on your RAID drive, you might want to manually work out the configuration of that partition to optimally work with the underlying RAID configuration. Key here is specifying the correct stride and stripe-width:

Chunk = 512 (output by or set via mdadm)
Block = 4 (ext4 dictated)
Devices = 2
Stride = 512 / 4 = 128
Stripe width = 2 * 128 = 256
Resulting in the formatting command sudo mkfs.ext4 -v -L nvmeRAID -m 1 -b 4096 -E stride=128,stripe-width=256 /dev/md0.

Wrapup

I hope this works out for you. It definitely took me more research, trial, and error than I would have liked to arrive at a functional approach. At this point I have been running the resulting setup for a month or two without the hint of an issue.

A Change of Gears

2017-12-28 00:00:00 +0100

Having completed my latest contract, working splendidly with Charles and Nicholas over at framebunker, we decided to extend our combined reach and potential by joining forces.

I got into contracting in the first place in order to find a new team. This year I had made it my mission to end that search and as it turned out, the team was much closer than expected.

What this means:

Contracting continues, now through framebunker. If you have a job that needs doing, give us a shout.
We enter into final production on Static Sky.
As Static Sky ramps down, we spin up new and exciting production – stay tuned for more updates on that.
I pick up the slack on the feed here as we launch a series of technical blog posts from framebunker.
Both Behave development and Game AI North organisation continue unaffected.

I am very excited about this change as it allows us all to be more productive, while maintaining a more stable work environment. I look forward to share more in 2018.

CoreObject

2017-07-04 00:00:00 +0200

I once built a talk with a collection of new hacks. This talk went through a couple of almost-delivered scenarios and with the new side projects I have since accumulated, scheduling gets difficult.

So I would rather share this stuff in post form now than stall for a possible eventuality. I have no idea how frequent I will be able to post, but here is one at least.

Challenge

Most Unity projects get spread across multiple scenes very quickly. This is advantageous not only for clearly segmented production, but also for tweaking of load times, streaming, and much more.

An unfortunate side effect of this is an increase of challenges in referencing objects in a live session, across those scenes. Desired targets include objects baked into specific scenes and specific instances created at runtime.

Common approaches to solving this include dependency injection and directory singletons – both of which work really great. Unfortunately they both introduce a level of indirection in the Unity serialisation system, making it impossible to track cross-scene dependencies with generic tools. This also usually adds a programming- or finicky book-keeping step to the process of introducing new conceptual instances.

If we could solve this challenge, we would end up with a clear dependency chain in the serialisation system, which would allow us to fully utilise the tracking and tooling benefits this yields, while greatly reducing the barrier to add new concepts and interact with existing ones.

Detour

With each scene a serialised island, there is one more (ignoring bundles) serialisation hub which all scenes can access: Assets.

Both the instance and its consumer can reference the same asset, which we can then use to facilitate a connection between the two. Further, with both referencing the same asset, we now have that dependency information in the serialisation system. Also, adding new conceptual instances is just a matter of creating a new asset, assigning it to either end.

Another bonus is easy listing of all conceptual instances by simply finding all assets of the type. You can demonstrate this via the object picker. Should you prefer a drop-down selector to the object picker, such would be a fairly straight forward piece of property drawer code. You could even utilise asset folder structure to define drop-down nesting.

Implementation

The base (5.3 tested) implementation of this needs three pieces:

A ScriptableObject type representing the conceptual instance as an asset.
- Example abstract, example concrete.
One MonoBehaviour registering a source.
- Example.
A serialized reference to the ScriptableObject type or a MonoBehaviour acting as a gateway to the instance – similar to a MeshFilter component.
- Example.

In the given example, the instances being passed around are just GameObjects. However in a real world scenario you may also want to create CoreObject<T> derivatives passing references to specific components. Not only does this facilitate quicker access to the components you need – it also naturally separates CoreObjects into categories.

A corresponding instance component could look a little something like this.

Variants

While this base functionality has plenty of utility, I have on occasion played around with variants on the concept. For one, the base idea passes a single-source instance producer to multiple consumers, but there is no reason why you could not use this setup to register/unregister to a list of instances in stead.

A personal favourite variant is more fundamentally different. In stead of facilitating references, the core objects wrap UnityEvents – turning them into global “game events”. This allows you to completely decouple event producers and consumers, while maintaining dependency tracking through the serialisation system.

The pattern I mostly use such events in start with the producers invoking the event via a serialised UnityEvent. Say you expose a “fire” UnityEvent to which you add a particle effect invocation, audio playback, and finally the invocation of the CoreGameEvent “gunshot”.

This “gunshot” event would then be referenced by CoreGameEventHandler components which also exposes a “handle” UnityEvent. The action list of this event could then include stuff like switching the soundscape, putting NPCs into an alert state, trigger an auto save, and whatever else might be useful.

Not only do you get decoupling of produces and consumers – you also get to centralise handling details around the individual consumers as opposed to leaving them all jumbled up at the producer.

So what do you think? Do you use something similar? What cool variants and tooling do you employ?

Source and examples

Unity Protocol Buffers

2017-02-17 00:00:00 +0100

Time for another “ah, crap – better put something up here before heading to GDC”-post. And since I did some re-tinkering with the integration of protobuf in Behave late last year, some sharing on that topic seemed in order.

So yes, stuff is happening with Behave – no abandonment has taken place. On the contrary I have been working on several, long-running, feature branches. I’ll get around to merging some of those in and doing another release. More on that soonish.

Anywho, last I checked, protobuf-net was still a more complete solution than the younger protobuf for C# one, so I still stick with protobuf-net and can therefore only talk about that here. However it looks like the latter is seeing more rapid development, so I’ll probably review my choice when next I need to tinker with the integration.

More interestingly, some of the initial people behind protobuf have since gone and built the new and shiny Cap’n Proto. This looks to be even more powerful than its predecessor, but at time of writing it is still not as mature or implementation-rich as protobuf. Critically for the context of this post, there is no C# implementation yet. Cool stuff, though – worth keeping an eye on.

Overview

So what is the Protocol Buffers project? Succinctly, it is a compact binary serialisation format with very fast cross-platform and -language implementations. Super useful when you need to quickly move some data between memory and some other location – a file / a network peer / whatever.

It was first designed, implemented, and maintained by Google for communications between network peers on their internal network. Since then, it has been implemented in a number of different languages – including the protobuf-net implementation, built and maintained by Marc Gravell.

Serialisation use in Behave has gone through a couple of distinct phases:

.net binary serialiser for asset.
protobuf-net for asset.
protobuf-net for asset and remote debugger protocol.
JSON for asset, protobuf-net for remote debugger protocol.

Additionally I’ve used protobuf-net for runtime saving or build pipelines on various client projects.

Aside from speed and compression, protobuf-net vs. the .net binary serialiser was also an escape of the frustrating lack of support for versioning or even simple structural refactors of serialised types. The switch to JSON for the .behave asset files of-course came when I decided that merge-ability could be a fun thing to support.

On top of this general migration, I also went through a couple of different protobuf-net integrations – as my use cases and requirements changed. It just so happens that this gave me full coverage of the three approaches supported by protobuf-net, so no need for extra research before writing this post. Win!

Integration

For serialisation to work, you need a a serialiser and a schema. Protobuf-net offers a couple of ways for you to provide those:

[Attribute] markup with runtime generated serialiser.
[Attribute] markup with pre-generated serialiser.
.proto file description with pre-generated serialiser.

I did this list in order of simplicity, which also happens to be the order in which I switched through them in the integration with Behave.

The first option is very .net esque and protobuf-net is indeed compatible both with its own serialisation attributes and the more general-purpose .net serialisation attributes. However since your interest here is in a Unity context, I’m sure that you have already spotted the problem with this approach.

Given that my initial use for protobuf-net was just for asset serialisation (which is editor-time only), I had no problem with the serialisation solution relying on JIT compilation in order to construct the serialiser at runtime. However as soon as I expanded my use case to include the runtime debugger, relying on JIT would mean not supporting the debugger on AOT platforms like iOS and consoles. Further, as Unity continues transitioning to their IL2CPP solution, you’re looking at a future where most will want to do AOT on all platforms.

So when introducing the remote debugger (previous debugger implementation was just 100% in-editor), I started to pre-generate the serialiser. This entails feeding your compiled assembly with [Attribute] markup to the protobuf-net precompile tool, which in turn generates an assembly with the serialiser type. For an example of how I used to do that, here’s a snippet of perl.

Extraction

Everything works! Win, right? Well… as I was expanding debugger support to non-C# targets and doing a general code cleanup, I got increasingly annoyed by having serialisation implementation detail in my general data type code. So I started looking at how protobuf-net supports the standard protobuf .proto format for schema definition.

As things stand, it takes a bit of work – specifically this work – to get going, but once there it is solid. In stead of using the precompile tool, you need to build and run the protogen tool from the protobuf-net repository. If you’re on Windows, things may just work straight out of the gate, but you may want to see what is in my patch anyway.

So how does this work? Well, the input is no longer [attribute] marked assemblies, but in stead a .proto definition file. You can find great detail on that in the general protobuf documentation, but you may want to consult the protobuf-net docs for implementation specifics/limits. Also, the output is not an assembly, but simply a C# file.

Keep in mind that protogen relies on the protoc tool from the general Google protobuf tools. I fetched this from homebrew as the “protobuf” package.

Again I have a snippet of perl to illustrate use – as well as an example .proto file. In this case the Behave debugger protocol definition.

Using this approach, I now have a nicely separated codebase and no duplication of schema definition between the C# debugger runtime and others. One trick remains though…

Object pooling

A mildly active Behave debugger session involves a lot of messages. I have very little interest in blowing up the garbage collector by constantly instantiating new messages and leaving them to get collected. So how do we integrate an object pool setup with the generated protobuf-net serialiser?

While not exactly as pretty as I would like, my answer is making use of the partialness of the generated types – in order to add a static pool and -constructor to hook it up.

And that’s pretty much what I’ve got. If I missed something or you have related questions, feel free to ping me and I’ll try to update this post when I can.

Behave 2.7

2016-02-13 00:00:00 +0100

The serialisation update is here! Ok maybe that is not the most exciting change for everyone, but I for one am quite happy about having finally shipped it. Aside from making life a lot simpler for anyone vaguely interested in collaborating on Behave libraries across version control systems, this lays the groundwork for a good number of runtime changes I have been wanting to do for a while.

Aside from just fixing up serialisation, this release focuses on improving editor usability with a good selection of tweaks and features. Some choice examples:

Behaviour tree minimap navigation.
Subtle indicator of tree origin direction when it is out of canvas view.
“Larger fonts” toggle for better visibility.
Hiding of component- and sidebar, plus a bunch of other canvas navigation tweaks.

Onward

After clearing this serialisation system obstruction out of the way, the road is now open for a host of planned features. Most of these are aimed at improving runtime flexibility, by increasing the available options for the tools you are already familiar with.

If you happen to be at this years GDC in a little months time, perhaps we could meet and exchange ideas on this?

Changelist:


2.7:
 - New library serialisation.
     - Backwards compatible with all 2.x variants.
     - All other formats are deprecated: text, binary, export/asset.
         - Exports are auto-upgraded, but please make sure to follow editor instructions when opening asset-type libraries.
         - Run your upgrades now. Future versions of Behave will not support deprecated serialisation.
     - Support for pre-2.0 decorators finally dropped.
     - Faster and less frequent serialisation.
     - Designed for optimal inter action with version control systems.
     - Custom inspector for Behave libraries is dropped in Unity 4.
         - Aside from the visuals, this just means that the Behave window now only can be launched by double-clicking the library or via the Window menu.
 - Behaviour tree minimap navigation - activate via alt+space.
 - Float/Clamp toggle for turning off canvas clamping when setting up large new tree structures.
 - Subtle indicator of tree origin direction when it is out of canvas view.
 - "Warn on non-mandatory actions and contexts" compiler toggle to support team communications.
 - "Larger fonts" toggle for better visibility.
 - Hiding of component- & sidebar.
 - Similar to reference components, components (and connections) referencing records can now also be double-clicked to navigate there.
 - Compiler adapted to new Unity distribution structure on OS X with Unity 5.4.
 - Home screen update.
 - Improved editor performance - particularly on larger libraries.
 - Better error handling for establishing debugger connections.
 - Tidying up and calming antivirus software by using Unity project Temp folder in stead of global temp.


2.6.4:
 - Fixed protobuf compatibility with Unity Windows.NET build targets.


2.6.3:
 - Fixed compiler execution on certain versions of OS X.
 - Minus one popup in the Windows compilation process.


2.6.2:
 - Fixed Windows compiler issue.


2.6.1:
 - Addressed Unity 5.1 compiler configuration compatibility issue.


2.6:
 - Removed backwards compatibility with Behave 1.x assets. That should be sufficient grace.
 - Removed a number of utility types from publicly accessible namespace - should be a lot less cluttered now.
 - Changed Tree.TickWrapping to now take exception handling behaviour parameter as an enum in stead of a boolean.
 - Changed Tree.LastTickTime to return DateTime ticks rather than a UnityEngine.Time.time value.
 - Fixed corner case where Unity could lock up while stripping Behave.
 - Fixed setting of Tree.LastTicked and Tree.ActiveID. Both will always reset to ActionType.Unknown, never -1.
 - Fixed compiler being able to handle Unity being installed on a path containing spaces.
 - Improved compiler error messages.
 - Added intellisense documentation to the runtime API.
 - Added int[] Tree.SupportedActions.
 - Added Tree.Owner for better handling of agent reassigning in pooling scenarios and similar.
 - Added ICollection. GetAsInt (int) and GetAsBool (int).
 - Action handler classes.
     - Added IActionClass and Tree.SetForwards (int, IActionClass) for init/tick/reset handling to a specific class instance.
     - Added Tree.ReflectForwards (GameObject) setting forwards for MonoBehaviours implementing IActionClass.
 - Editor polish.
     - "Now Behave library being edited" label now comes with a shortcut "Create" button.
     - More informative load messages in the home screen, including progress bars.
     - With Unity 5 dropping the engine feature delta, in-editor video viewing is now available on non-pro licenses as well.


2.5.4:
 - Address Unity 5 compatibility issue with the home screen.


2.5.3:
 - Fix issue with home screen not closing.


2.5.2:
 - Play nice on Unity free again.


2.5.1:
 - More rigid network code for setting up the debugger.
 - Upgraded protobuf-net distribution to r668, Unity, full.
 - Drowned out debug logging accidentally included in 2.5.0.


2.5:
 - Optional text serialisation.
     - Switch between text and binary serialisation in library settings.
     - Full and partial exports to text or binary file.
     - NOTE: Libraries opened in this version of Behave will be upgraded and are afterwards not backwards compatible.
 - Compiler upgrade.
     - Added lots of little CPU and memory optimisations.
     - Added tree analysis leading to larger memory saves on references when possible.
     - Improved handling of circular references.
     - Improved error reporting.
     - Changed name validation.
         - Underscores are no longer allowed in names - in order to avoid name clashes with pathed assets.
         - Fixed previously allowing leading numbers in names.
 - Runtime API.
     - Added library.ReflectForwards (ICollection, object).
     - Added library.SupportedRecords (ICollection).
     - Added blueprint.SupportedTrees.
     - Added Library.DidInstantiate (ICollection / Tree).
     - Changed generated enums to be valued continuously.
     - Changed get calls for unhandled records to now return -1.
 - Editor polish.
     - Improved debugger view.
         - Improved control bar layout.
         - Improved rendering of active referenced trees.
         - Fixes to follow-focus.
         - Debug view no longer re-crops to current visible tree area, preventing the previous behaviour of the view jumping around even without follow-focus enabled.
         - Improved sidebar listing of available- and connected runtimes.
         - Improved handling of disconnecting runtimes.
         - Improved tracking of available runtimes. No more need for refreshing.
         - Improved UI for adding debugger overrides.
         - Added "Add overrides enabled" setting (on by default).
     - Library vertical scrollbar replaced by standalone scroll buttons top / bottom when relevant - reducing visual noise.
     - Smaller tweaks and fixes.
 - Improved internal test coverage.


2.4.2:
 - Fixed some issues related to handling priority selectors in the recently update debugger.


2.4.1:
 - Fix corner case when restructuring library assets.


2.4:
 - Introducing “records”.
     - Float fields stored in “collections” - essentially ‘smart tables’.
     - Blackboard fields.
     - Utility curves.
     - Utility graphs.
     - Behaviour Tree integration.
          - “Tester” component.
          - “Writer” component.
          - Record mode for Priority Selector component.
 - Editor behavioural change: Deleting a tree no longer deletes reference nodes targeting that tree from other trees. Their target is merely cleared.
 - Editor behavioural change: The Reference node is only available in the component bar if the library contains more than one tree. Same behaviour for Tester and Writer with records.
 - Behavioural change: Invert flag is now properly applied by the Priority Selector.
 - Debugger connection stability improvement.
 - Connection disabling has been removed. This confusing feature caused more harm than good.
 - Video player now displays video title and description.


2.3.5:
 - Fix home screen video player.


2.3.4:
 - Better error messages in case of insufficient file access privileges (OHAI Perforce).
 - Added simple version check to the home screen.
 - New icons for Priority Selector and "form new tree".
 - Smaller, embedded, JSON library.
 - Fix annoying behaviour with tree renaming where text fields would keep last entry.
 - Fix similar behaviour with component text fields on tree selection change.
 - Unity 5 compatibility fixes (tested on beta 2).


2.3.3:
 - Fix focus handling issue with auto-complete.
 - Parametrisation is now correctly applied for reset handler calls.


2.3.2:
 - Fix sidebar item duplication and deletion only functional during rename.


2.3.1:
 - Fix GUI error being logged on tree selection.
 - Proper handling of currently edited library asset being deleted.


2.3:
 - Added support for instant repeaters.
 - Added support for mandatory contexts.
 - Added optional target parameter to Tree.ResetForwards.
 - Added Tree.TickWrapping method.
 - Added debugger auto-launch settings to library build options.
 - Added home screen - shown on launch or via the Help menu, full of resources.
 - Behavioural change: Decorators no longer forward the instant flag of their child node.
 - Behaviour of decorators receiving a Running signal from their control connection is now defined.
 - Dependencies now shipped as separate (replaceable) files for minimal project impact.
 - Editor polish.
     - Added support for running multiple editor debugging instances on one machine.
     - Added focus-follow for debugger and canvas.
     - Added both scrollwheel/pad and drag-to-scroll support to all canvas views.
     - Drag connection to occupied input slot to replace.
     - Directly drag-drop trees from the sidebar onto the canvas as reference nodes.
     - Instant flag doesn't show on nodes connected to parents ignoring the flag.
     - "Last edited library" now tracked per project.
     - Last selected tree is now also restored between sessions and playmode transitions.
     - A bunch of minor adjustments and improvements.
     - Performance + memory + disk space optimisations.
     - Minor bugfixes.
 - New documentation layout and enhancements (already released as available).
 - Reshuffling of code to make way for exciting new things down the road.


2.2.2:
 - Addressed protobuf issue causing Parallel component completion setting to not get stored correctly.


2.2.1:
 - Fixed issue causing parallel components to fail compilation.


2.2:
 - Behavioural change: Support for multiple different action handler signatures is discontinued effective immediately.
    - Generated agent blueprint as well as accepted reflected signatures are now only:
        - public BehaveResult [Tick/Init][Name]Action (Tree sender);
        - public void Reset[Name]Action (Tree sender);
 - Added Tree class accessors - all valid only during an active init/tick/reset handler call:
    - string ActiveStringParameter
    - float ActiveFloatParameter
    - IAgent ActiveAgent
    - object ActiveData
 - Rather than waiting for the Unity bugfix, the built-in compiler API is now bypassed in favour of direct compiler binary execution.
    - An additional system-level mono SDK install is no longer needed to work around this Unity bug.
 - Tree.SetForward now returns boolean success value rather than logging error when passed unused action.
 - Full-library export now also available from the library settings in stead of menu-only access.
 - Even more UI polish: Better word-wrap, more contrasts for emphasis, instant flag hidden in unaffected scenarios and much more.
 - "Form new tree" nicely moves the new subtree up around the root socket rather than retaining the old position.
 - Default frame-rate of DebugUpdater now set to 20 fps rather than every frame. Gun safety and all that.
 - Massive codegen size save for larger libraries - from optimisation of tree-agent introspection codegen. Compile-time win.
 - More rigid parameter evaluation (remove needless occasional extra codegen).
 - Fixed copy and duplication procedures incorrectly not copying parameter values.
 - Force Unity to write serialised data before compiling.


2.1.1:
 - Fixed issue with tree paramterisation breaking debugger visualisation.


2.1:
 - Added autocomplete on action names, string parameters and component contexts.
 - Added mandatory actions (present in library regardless of use in trees).
 - Added partial library export and library merging / import-to-library.
 - Added Behave stripping - leave behind only a very lean Behave runtime, for redistribution or super slim deploy.
 - Added parameterised trees - get even more re-use out of your trees by overriding context and variables per reference.
 - Behavioural change: Tree.Reset will now delay resetting tree variables until after complete component reset.
 - Behavioural change: Tree.Tick will no longer automatically reset the tree upon completion. This plays very well with:
 - Added Tree.TickContinuously (int iterationCap = 1024) - ticks tree till completion or iteration cap is reached.
 - Added BehaveResult Tree.Result (bool value) utility function to side with Tree.Invert (BehaveResult value).
 - Added int Debugging.Local.ConnectionCount property - facilitating debug behaviour dependent on Behave debugger presence.
 - Moved default install path from Assets/Behave to Assets/Plugins/Behave.
    - WARNING: See Readme.txt upgrade guide.
 - Fixed decorator control node removal resulting in detached control connection.
 - Fixed decorator incorrectly not forwarding the instant status of its child node.
 - Fixed scenario where agent classes would get reflected for handlers even if blueprint derived.
 - Fixed unexpected corner cases of bad library configuration failing compiles (now causing warnings or simply being ignored).
 - Fixed Behave window breaking down when exiting play mode after having been active there.
 - Fixes to context getting incorrectly reset in some scenarios.
 - Fixed copy / paste functionality missing in some text fields.
 - Lots of UI polish and minor fixes.


2.0:
 - New editor.
    - Unified all controls into one window.
    - Completely customised look - compatible with both Unity light and dark skin.
    - Keyboard authoring interface in addition to drag & drop.
 - Debugger upgrade.
    - Remote debugging.
    - Tick and reset.
    - "Unplug" for debugger-only control.
    - Breakpoints.
    - Action overrides - override the return values of specified actions.
 - Decorator rewrite.
    - Deprecated old decorator.
       - Still compiles, but cannot be added.
    - Decorators are now control-type components.
       - No specific decorator handlers on agents.
       - Control is achieved via third connection socket.
    - Two initial decorator types: Repeater and Interrupter.
 - Asset restructuring.
 	- No more collections - for nesting, use paths in tree names.
 	- New serialization format - see Readme.txt for upgrade instructions.
    - Collections are gone. Use Unix-style path names for tree organisation.
 - IAgent.Tick (Tree, bool) changed to IAgent.Tick (Tree)
	- Init calls can now only be handled explicitly
 - Full coverage online documentation and other community resources.
 - Countless fixes and polish items.

1.5:
 - Added Tree.TickActive - indicating if a tree is currently executing a tick or if it has completed evaluation.
 - In-tick tree reset requests now result in a reset at the end of tree evaluation rather than an immediate one.
 - Removed the legacy compatibility assembly (for compatibility with pre-1.2 assets) from the default distribution.
 - Fixed "X must be instantiated using the ScriptableObject.CreateInstance method instead of new X" warnings.

1.4:
 - Fixed decorator incorrectly initializing on every returning tick when tick handler returned Running.
 - Fixed debugger view dependency on active tree editor - causing reported null reference exception.
 - Fixed missing scrollbars on browser window.
 - Reduced logging noise from debug builds.
 - Agent blueprints can now be MonoBehaviour based.
 - Clarified wording on "library not loaded".
 - Now handling reset a bit cleverer - potential performance boost.
 - Changed the default success criteria of Parallel components from SuccessOrFailure to Success.
 - Added "instant" flag to components. Components marked instant will affect sequences and selectors when moving between child nodes. When a child node marked "instant" completes, rather than waiting for the next tree tick to tick the following child node, the sequence or selector does it immediately.
 - The compiler progress bar returns.
 - Editing a Behave asset now focuses the asset browser last instead of the tree editor.
 - Behave can now be installed in any subfolder of Assets - provided its internal folder structure remains intact.
 - Compilation speed improvements.
 - Enabled references across collection borders.
 - Added Tree.DataSize - returning the number of bytes used by a tree instance. This is also displayed in the debugger window.

1.3:
 - Fixed priority selector init flag resetting.
 - Fixed reference handling on tree renaming and deletion.
 - Fixed references still pointing at old collection post collection duplication.
 - Fixed compilation of actions and decorators with non-integral float parameters.
 - Fixed GUI list behaviours.
 - Fixed passing context back and forth between referenced trees and referrer.
 - Fixed reordering connections not causing save state to be set.
 - Various GUI tweaks.
 - Runtime optimisations.
 - Enabled scrolling of the tree list in the debugger window.
 - Exposed Tree.ReflectForwards.
 - Added Tree.ResetForwards.
 - Added agent blueprints - connecting to agent handlers through virtual methods rather than reflected handlers.
 - Made connections selectable in the tree editor via the connection nub.
 - Moved connection managing from the inspector to the tree editor, where it is more visible and handy.
 - Added support for disabling connections - for faster, less destructive debugging.
 - Double-click reference component to jump to referenced tree.
 - Changed "Cannot update unregistered tree" handling from an exception to an error message. Still need more info.
 - Now showing string and float parameters in the tree editor if either is set.

1.2:
 - Fixed release building.
 - Added the priority selector, introducing the API points:
    - int IAgent.SelectTopPriority (Tree sender, params int[] IDs);
    - enum BL[LibraryName].PriorityType { PriorityName, ..., Unknown };
 - Added support for branch contexts, introducing the API points:
    - enum BL[LibraryName].ContextType { ContextName, ..., Unknown };
    - int Tree.ActiveContext { get; }
 - Changed IAgent interface:
    - Tick (bool init, Tree sender) -> Tick (Tree sender, bool init)
 - Added "Select asset" button to the standard inspector.
 - Interface polish.

1.1:
 - Made compatible with Unity 3.0 - including ensuring that the 2.5-2.6.1 bug preventing compilation on Windows is fixed on the Unity side.
 - UI tweaks and polish.
 - Added the option to have return values of nodes be inverted.
 - Fixed compiler bug when facing sequence or selector with only one child node.
 - Renamed Library.Tree -> Library.InstantiateTree.
 - Added Tree.Tick () and Tree.Reset () methods using the IAgent instance given at instantiation of the tree.
 - Removed the abstract Library class.
 - Changed generated library class:
    - Trees -> TreeType.
    - Actions -> ActionType.
    - Decorators -> DecoratorType.
    - Tree InstantiateTree (int, IAgent) -> static Tree InstantiateTree (TreeType, IAgent)
    - int TreeID (Tree) -> static TreeType Type (Tree)
    - bool IsAction (int) -> static bool IsAction (int)
    - bool IsDecorator (int) -> static bool IsDecorator (int)
 - Upped editor performance.
 - New asset structure change with automatic import of Behave 1.0 assets.
 - Changed action and decorator handlers. These are now the handlers mapped:
    - public BehaveResult Init[Name][Action/Decorator] (Tree sender, string stringParameter, float floatParameter, IAgent agent, object data);
    - public BehaveResult Tick[Name][Action/Decorator] (Tree sender, string stringParameter, float floatParameter, IAgent agent, object data);
    - public void Reset[Name][Action/Decorator] (Tree sender, string stringParameter, float floatParameter, IAgent agent, object data);
    - public BehaveResult Init[Name][Action/Decorator] (Tree sender);
    - public BehaveResult Tick[Name][Action/Decorator] (Tree sender);
    - public void Reset[Name][Action/Decorator] (Tree sender);
    - public BehaveResult [Name][Action/Decorator] {get; set;}
 - Added support for collection duplication.
 - Moved debug rendering into the editor with its own dedicated debugger window, including actor listing.
 - Debugger now renders active sub-trees as well.

1.0:
 - Lots o fixes for Unity 2.5 compatibility.
 - Closer editor integration with new UI.
    - Component bar.
    - Node inspection.
 - Runtime tree debugging.
 - Direct mapping of action and delegate handlers via reflection.
 - Kitty demo.

0.3b:
 - Various tweaks to the editor UI.
 - Optimised the compiler.
 - Fixed the following editor bugs from the bug list:
    - When the editor scrips are recompiled and when the user enters and exits play mode, running editors will loose references, causing undetermined behaviour.
    - When a compile fails, the Behave compiler is locked and unusable – requiring a unity relaunch.
 - Implemented the following wishlist items:
    - Behaviour tree decorators.
    - Stand-alone edition of the Behave editor.
    - Behaviour tree debugging features.

0.2b:
 - Made the demo project a whole lot more interesting.
 - Implemented the editor wish list item: Ability to move trees between collections.
 - Various tweaks to the editor UI.
 - Fixed a ton of bugs in the compiler.
 - Fixed the following editor bugs from the bug list:
    - Not able to delete collections from library.
    - Not able to delete trees from library.
    - Not able to rename collections from library.
    - Not able to rename trees from library.
    - Missing "tree has been changed, save changes?" dialog.

0.1b:
 - Some documentation is available.
 - Compiler is now creating sensible output.
 - A simple test tree and script using it is provided with the demo.
 - Documentation is linked directly from the unity help menu.
 - Less spamming of the log with debug information.

Pre-release:
 - Magic happened. Behave was born.

Learn

2015-07-17 00:00:00 +0200

I’m just about to wrap up a weeks vacation before it’s back on the horse again – as I leave for the nucl.ai conference Sunday morning. If you’re going, maybe I’ll see you there? If not, you should go check out the live stream.

Basis

Anyway, so in my vacation I built a little project when I could find a bit of time here and there. It was inspired by work on a client project earlier in the year, where we looked at wiring up a neural network for analysis of a noisy sensor.

The problem at the time was that although I could find a few C# NN libraries here and there, they were either not really a good fit or had a not-so-open open-source license like GPL (good thing I don’t have comments enabled here, eh?).

Anyway, that felt a bit silly, so I figured that I’d at least get the ball rolling on a somewhat sane, MIT licensed, alternative. Either that or just have some fun with it until someone would point me at the obvious alternative solution, which I had overlooked in my earlier search.

Project

In its initial shape, the library just lets you construct a layered network, randomize its weigths, train it with a straight forward implementation of back-propagation with adjustable learning rate and momentum, and of-course run it.

There is no serialisation interface yet, I’ll probably want to do a simplification pass over it, and I have a few edge case tests I’d like to run. However all of that is noted on the project TODO and shouldn’t amount to all that much work – in case you fancy taking a stab at it.

I did set up a little Unity project for testing though (the library itself does not depend on anything in Unity). It has a few debug views, a test scenario of the XOR case, and a curve visualisation of training error. Look, I uploaded several pretty pictures:

Access

Feel like giving it a go? As mentioned, the license is MIT, so you can really do whatever with it. However should you want to contribute, pull requests are most welcome. I don’t know how much time I can spare on this next to contracts and Behave, but at least it was a fun thing to work-on-while-not-working.

Whatever the case, this is where you can find the repositories:

Behave 2.6

2015-06-13 00:00:00 +0200

Still plenty of open feature branches on Behave, but now is the time for the next drop. With a little over a week till Unite Europe and shortly thereafter nucl.ai (formerly GameAIconf), I’m also heading into a bit of a plenty-to-do period. Perhaps we’ll meet at the conference and have a chat about it?

With only minor editor tweaks, the 2.6 release of Behave focuses primarily on API changes and added capabilities. The more exciting of these is probably the addition of action classes, allowing you to implement actions not just as singular event handlers on agent classes, but as standalone classes – or in the case of Unity, as components.

The reference documentation has been updated with a “Composition” section describing the setup in more detail.

Do note that in this release, Behave is no longer backwards compatible with Behave 1.x assets.

Onwards

Unfortunately some planned features are taking a bit longer as they depend on a serialisation system refactor, requiring an extended period of focus. The groundwork for this has been effectuated and given time, should shoulder an effective change leading to interesting new capabilities.

Changelist:


2.6:
 - Removed backwards compatibility with Behave 1.x assets. That should be sufficient grace.
 - Removed a number of utility types from publicly accessible namespace - should be a lot less cluttered now.
 - Changed Tree.TickWrapping to now take exception handling behaviour parameter as an enum in stead of a boolean.
 - Changed Tree.LastTickTime to return DateTime ticks rather than a UnityEngine.Time.time value.
 - Fixed corner case where Unity could lock up while stripping Behave.
 - Fixed setting of Tree.LastTicked and Tree.ActiveID. Both will always reset to ActionType.Unknown, never -1.
 - Fixed compiler being able to handle Unity being installed on a path containing spaces.
 - Improved compiler error messages.
 - Added intellisense documentation to the runtime API.
 - Added int[] Tree.SupportedActions.
 - Added Tree.Owner for better handling of agent reassigning in pooling scenarios and similar.
 - Added ICollection. GetAsInt (int) and GetAsBool (int).
 - Action handler classes.
     - Added IActionClass and Tree.SetForwards (int, IActionClass) for init/tick/reset handling to a specific class instance.
     - Added Tree.ReflectForwards (GameObject) setting forwards for MonoBehaviours implementing IActionClass.
 - Editor polish.
     - "Now Behave library being edited" label now comes with a shortcut "Create" button.
     - More informative load messages in the home screen, including progress bars.
     - With Unity 5 dropping the engine feature delta, in-editor video viewing is now available on non-pro licenses as well.


2.5.4:
 - Address Unity 5 compatibility issue with the home screen.


2.5.3:
 - Fix issue with home screen not closing.


2.5.2:
 - Play nice on Unity free again.


2.5.1:
 - More rigid network code for setting up the debugger.
 - Upgraded protobuf-net distribution to r668, Unity, full.
 - Drowned out debug logging accidentally included in 2.5.0.


2.5:
 - Optional text serialisation.
     - Switch between text and binary serialisation in library settings.
     - Full and partial exports to text or binary file.
     - NOTE: Libraries opened in this version of Behave will be upgraded and are afterwards not backwards compatible.
 - Compiler upgrade.
     - Added lots of little CPU and memory optimisations.
     - Added tree analysis leading to larger memory saves on references when possible.
     - Improved handling of circular references.
     - Improved error reporting.
     - Changed name validation.
         - Underscores are no longer allowed in names - in order to avoid name clashes with pathed assets.
         - Fixed previously allowing leading numbers in names.
 - Runtime API.
     - Added library.ReflectForwards (ICollection, object).
     - Added library.SupportedRecords (ICollection).
     - Added blueprint.SupportedTrees.
     - Added Library.DidInstantiate (ICollection / Tree).
     - Changed generated enums to be valued continuously.
     - Changed get calls for unhandled records to now return -1.
 - Editor polish.
     - Improved debugger view.
         - Improved control bar layout.
         - Improved rendering of active referenced trees.
         - Fixes to follow-focus.
         - Debug view no longer re-crops to current visible tree area, preventing the previous behaviour of the view jumping around even without follow-focus enabled.
         - Improved sidebar listing of available- and connected runtimes.
         - Improved handling of disconnecting runtimes.
         - Improved tracking of available runtimes. No more need for refreshing.
         - Improved UI for adding debugger overrides.
         - Added "Add overrides enabled" setting (on by default).
     - Library vertical scrollbar replaced by standalone scroll buttons top / bottom when relevant - reducing visual noise.
     - Smaller tweaks and fixes.
 - Improved internal test coverage.


2.4.2:
 - Fixed some issues related to handling priority selectors in the recently update debugger.


2.4.1:
 - Fix corner case when restructuring library assets.


2.4:
 - Introducing “records”.
     - Float fields stored in “collections” - essentially ‘smart tables’.
     - Blackboard fields.
     - Utility curves.
     - Utility graphs.
     - Behaviour Tree integration.
          - “Tester” component.
          - “Writer” component.
          - Record mode for Priority Selector component.
 - Editor behavioural change: Deleting a tree no longer deletes reference nodes targeting that tree from other trees. Their target is merely cleared.
 - Editor behavioural change: The Reference node is only available in the component bar if the library contains more than one tree. Same behaviour for Tester and Writer with records.
 - Behavioural change: Invert flag is now properly applied by the Priority Selector.
 - Debugger connection stability improvement.
 - Connection disabling has been removed. This confusing feature caused more harm than good.
 - Video player now displays video title and description.


2.3.5:
 - Fix home screen video player.


2.3.4:
 - Better error messages in case of insufficient file access privileges (OHAI Perforce).
 - Added simple version check to the home screen.
 - New icons for Priority Selector and "form new tree".
 - Smaller, embedded, JSON library.
 - Fix annoying behaviour with tree renaming where text fields would keep last entry.
 - Fix similar behaviour with component text fields on tree selection change.
 - Unity 5 compatibility fixes (tested on beta 2).


2.3.3:
 - Fix focus handling issue with auto-complete.
 - Parametrisation is now correctly applied for reset handler calls.


2.3.2:
 - Fix sidebar item duplication and deletion only functional during rename.


2.3.1:
 - Fix GUI error being logged on tree selection.
 - Proper handling of currently edited library asset being deleted.


2.3:
 - Added support for instant repeaters.
 - Added support for mandatory contexts.
 - Added optional target parameter to Tree.ResetForwards.
 - Added Tree.TickWrapping method.
 - Added debugger auto-launch settings to library build options.
 - Added home screen - shown on launch or via the Help menu, full of resources.
 - Behavioural change: Decorators no longer forward the instant flag of their child node.
 - Behaviour of decorators receiving a Running signal from their control connection is now defined.
 - Dependencies now shipped as separate (replaceable) files for minimal project impact.
 - Editor polish.
     - Added support for running multiple editor debugging instances on one machine.
     - Added focus-follow for debugger and canvas.
     - Added both scrollwheel/pad and drag-to-scroll support to all canvas views.
     - Drag connection to occupied input slot to replace.
     - Directly drag-drop trees from the sidebar onto the canvas as reference nodes.
     - Instant flag doesn't show on nodes connected to parents ignoring the flag.
     - "Last edited library" now tracked per project.
     - Last selected tree is now also restored between sessions and playmode transitions.
     - A bunch of minor adjustments and improvements.
     - Performance + memory + disk space optimisations.
     - Minor bugfixes.
 - New documentation layout and enhancements (already released as available).
 - Reshuffling of code to make way for exciting new things down the road.


2.2.2:
 - Addressed protobuf issue causing Parallel component completion setting to not get stored correctly.


2.2.1:
 - Fixed issue causing parallel components to fail compilation.


2.2:
 - Behavioural change: Support for multiple different action handler signatures is discontinued effective immediately.
    - Generated agent blueprint as well as accepted reflected signatures are now only:
        - public BehaveResult [Tick/Init][Name]Action (Tree sender);
        - public void Reset[Name]Action (Tree sender);
 - Added Tree class accessors - all valid only during an active init/tick/reset handler call:
    - string ActiveStringParameter
    - float ActiveFloatParameter
    - IAgent ActiveAgent
    - object ActiveData
 - Rather than waiting for the Unity bugfix, the built-in compiler API is now bypassed in favour of direct compiler binary execution.
    - An additional system-level mono SDK install is no longer needed to work around this Unity bug.
 - Tree.SetForward now returns boolean success value rather than logging error when passed unused action.
 - Full-library export now also available from the library settings in stead of menu-only access.
 - Even more UI polish: Better word-wrap, more contrasts for emphasis, instant flag hidden in unaffected scenarios and much more.
 - "Form new tree" nicely moves the new subtree up around the root socket rather than retaining the old position.
 - Default frame-rate of DebugUpdater now set to 20 fps rather than every frame. Gun safety and all that.
 - Massive codegen size save for larger libraries - from optimisation of tree-agent introspection codegen. Compile-time win.
 - More rigid parameter evaluation (remove needless occasional extra codegen).
 - Fixed copy and duplication procedures incorrectly not copying parameter values.
 - Force Unity to write serialised data before compiling.


2.1.1:
 - Fixed issue with tree paramterisation breaking debugger visualisation.


2.1:
 - Added autocomplete on action names, string parameters and component contexts.
 - Added mandatory actions (present in library regardless of use in trees).
 - Added partial library export and library merging / import-to-library.
 - Added Behave stripping - leave behind only a very lean Behave runtime, for redistribution or super slim deploy.
 - Added parameterised trees - get even more re-use out of your trees by overriding context and variables per reference.
 - Behavioural change: Tree.Reset will now delay resetting tree variables until after complete component reset.
 - Behavioural change: Tree.Tick will no longer automatically reset the tree upon completion. This plays very well with:
 - Added Tree.TickContinuously (int iterationCap = 1024) - ticks tree till completion or iteration cap is reached.
 - Added BehaveResult Tree.Result (bool value) utility function to side with Tree.Invert (BehaveResult value).
 - Added int Debugging.Local.ConnectionCount property - facilitating debug behaviour dependent on Behave debugger presence.
 - Moved default install path from Assets/Behave to Assets/Plugins/Behave.
    - WARNING: See Readme.txt upgrade guide.
 - Fixed decorator control node removal resulting in detached control connection.
 - Fixed decorator incorrectly not forwarding the instant status of its child node.
 - Fixed scenario where agent classes would get reflected for handlers even if blueprint derived.
 - Fixed unexpected corner cases of bad library configuration failing compiles (now causing warnings or simply being ignored).
 - Fixed Behave window breaking down when exiting play mode after having been active there.
 - Fixes to context getting incorrectly reset in some scenarios.
 - Fixed copy / paste functionality missing in some text fields.
 - Lots of UI polish and minor fixes.


2.0:
 - New editor.
    - Unified all controls into one window.
    - Completely customised look - compatible with both Unity light and dark skin.
    - Keyboard authoring interface in addition to drag & drop.
 - Debugger upgrade.
    - Remote debugging.
    - Tick and reset.
    - "Unplug" for debugger-only control.
    - Breakpoints.
    - Action overrides - override the return values of specified actions.
 - Decorator rewrite.
    - Deprecated old decorator.
       - Still compiles, but cannot be added.
    - Decorators are now control-type components.
       - No specific decorator handlers on agents.
       - Control is achieved via third connection socket.
    - Two initial decorator types: Repeater and Interrupter.
 - Asset restructuring.
 	- No more collections - for nesting, use paths in tree names.
 	- New serialization format - see Readme.txt for upgrade instructions.
    - Collections are gone. Use Unix-style path names for tree organisation.
 - IAgent.Tick (Tree, bool) changed to IAgent.Tick (Tree)
	- Init calls can now only be handled explicitly
 - Full coverage online documentation and other community resources.
 - Countless fixes and polish items.

1.5:
 - Added Tree.TickActive - indicating if a tree is currently executing a tick or if it has completed evaluation.
 - In-tick tree reset requests now result in a reset at the end of tree evaluation rather than an immediate one.
 - Removed the legacy compatibility assembly (for compatibility with pre-1.2 assets) from the default distribution.
 - Fixed "X must be instantiated using the ScriptableObject.CreateInstance method instead of new X" warnings.

1.4:
 - Fixed decorator incorrectly initializing on every returning tick when tick handler returned Running.
 - Fixed debugger view dependency on active tree editor - causing reported null reference exception.
 - Fixed missing scrollbars on browser window.
 - Reduced logging noise from debug builds.
 - Agent blueprints can now be MonoBehaviour based.
 - Clarified wording on "library not loaded".
 - Now handling reset a bit cleverer - potential performance boost.
 - Changed the default success criteria of Parallel components from SuccessOrFailure to Success.
 - Added "instant" flag to components. Components marked instant will affect sequences and selectors when moving between child nodes. When a child node marked "instant" completes, rather than waiting for the next tree tick to tick the following child node, the sequence or selector does it immediately.
 - The compiler progress bar returns.
 - Editing a Behave asset now focuses the asset browser last instead of the tree editor.
 - Behave can now be installed in any subfolder of Assets - provided its internal folder structure remains intact.
 - Compilation speed improvements.
 - Enabled references across collection borders.
 - Added Tree.DataSize - returning the number of bytes used by a tree instance. This is also displayed in the debugger window.

1.3:
 - Fixed priority selector init flag resetting.
 - Fixed reference handling on tree renaming and deletion.
 - Fixed references still pointing at old collection post collection duplication.
 - Fixed compilation of actions and decorators with non-integral float parameters.
 - Fixed GUI list behaviours.
 - Fixed passing context back and forth between referenced trees and referrer.
 - Fixed reordering connections not causing save state to be set.
 - Various GUI tweaks.
 - Runtime optimisations.
 - Enabled scrolling of the tree list in the debugger window.
 - Exposed Tree.ReflectForwards.
 - Added Tree.ResetForwards.
 - Added agent blueprints - connecting to agent handlers through virtual methods rather than reflected handlers.
 - Made connections selectable in the tree editor via the connection nub.
 - Moved connection managing from the inspector to the tree editor, where it is more visible and handy.
 - Added support for disabling connections - for faster, less destructive debugging.
 - Double-click reference component to jump to referenced tree.
 - Changed "Cannot update unregistered tree" handling from an exception to an error message. Still need more info.
 - Now showing string and float parameters in the tree editor if either is set.

1.2:
 - Fixed release building.
 - Added the priority selector, introducing the API points:
    - int IAgent.SelectTopPriority (Tree sender, params int[] IDs);
    - enum BL[LibraryName].PriorityType { PriorityName, ..., Unknown };
 - Added support for branch contexts, introducing the API points:
    - enum BL[LibraryName].ContextType { ContextName, ..., Unknown };
    - int Tree.ActiveContext { get; }
 - Changed IAgent interface:
    - Tick (bool init, Tree sender) -> Tick (Tree sender, bool init)
 - Added "Select asset" button to the standard inspector.
 - Interface polish.

1.1:
 - Made compatible with Unity 3.0 - including ensuring that the 2.5-2.6.1 bug preventing compilation on Windows is fixed on the Unity side.
 - UI tweaks and polish.
 - Added the option to have return values of nodes be inverted.
 - Fixed compiler bug when facing sequence or selector with only one child node.
 - Renamed Library.Tree -> Library.InstantiateTree.
 - Added Tree.Tick () and Tree.Reset () methods using the IAgent instance given at instantiation of the tree.
 - Removed the abstract Library class.
 - Changed generated library class:
    - Trees -> TreeType.
    - Actions -> ActionType.
    - Decorators -> DecoratorType.
    - Tree InstantiateTree (int, IAgent) -> static Tree InstantiateTree (TreeType, IAgent)
    - int TreeID (Tree) -> static TreeType Type (Tree)
    - bool IsAction (int) -> static bool IsAction (int)
    - bool IsDecorator (int) -> static bool IsDecorator (int)
 - Upped editor performance.
 - New asset structure change with automatic import of Behave 1.0 assets.
 - Changed action and decorator handlers. These are now the handlers mapped:
    - public BehaveResult Init[Name][Action/Decorator] (Tree sender, string stringParameter, float floatParameter, IAgent agent, object data);
    - public BehaveResult Tick[Name][Action/Decorator] (Tree sender, string stringParameter, float floatParameter, IAgent agent, object data);
    - public void Reset[Name][Action/Decorator] (Tree sender, string stringParameter, float floatParameter, IAgent agent, object data);
    - public BehaveResult Init[Name][Action/Decorator] (Tree sender);
    - public BehaveResult Tick[Name][Action/Decorator] (Tree sender);
    - public void Reset[Name][Action/Decorator] (Tree sender);
    - public BehaveResult [Name][Action/Decorator] {get; set;}
 - Added support for collection duplication.
 - Moved debug rendering into the editor with its own dedicated debugger window, including actor listing.
 - Debugger now renders active sub-trees as well.

1.0:
 - Lots o fixes for Unity 2.5 compatibility.
 - Closer editor integration with new UI.
    - Component bar.
    - Node inspection.
 - Runtime tree debugging.
 - Direct mapping of action and delegate handlers via reflection.
 - Kitty demo.

0.3b:
 - Various tweaks to the editor UI.
 - Optimised the compiler.
 - Fixed the following editor bugs from the bug list:
    - When the editor scrips are recompiled and when the user enters and exits play mode, running editors will loose references, causing undetermined behaviour.
    - When a compile fails, the Behave compiler is locked and unusable – requiring a unity relaunch.
 - Implemented the following wishlist items:
    - Behaviour tree decorators.
    - Stand-alone edition of the Behave editor.
    - Behaviour tree debugging features.

0.2b:
 - Made the demo project a whole lot more interesting.
 - Implemented the editor wish list item: Ability to move trees between collections.
 - Various tweaks to the editor UI.
 - Fixed a ton of bugs in the compiler.
 - Fixed the following editor bugs from the bug list:
    - Not able to delete collections from library.
    - Not able to delete trees from library.
    - Not able to rename collections from library.
    - Not able to rename trees from library.
    - Missing "tree has been changed, save changes?" dialog.

0.1b:
 - Some documentation is available.
 - Compiler is now creating sensible output.
 - A simple test tree and script using it is provided with the demo.
 - Documentation is linked directly from the unity help menu.
 - Less spamming of the log with debug information.

Pre-release:
 - Magic happened. Behave was born.

Trusted Gear

2015-02-26 00:00:00 +0100

Yay! GDC is next week! Will I see you there?

Ahem. Straying a bit from my usual theme here of writing about whatever little snippet or gadget I’ve come up with, for this post I will in stead share some detail on recent changes to my tool-chain of choice, which I’m quite happy about.

Every craftsman has his or her opinion on tooling, but just about everyone share the desire to tune their toolbox. Even if your opinion / taste / whatever does not align with mine, I hope that you find something useful in here.

If you really had your sights on getting some funky code snippet out of this post, here’s an unholy perl module for firing off native prompt dialogs on OS X and Windows:

Unholy perl

And with that out of the way, let’s get something else sorted: This post will likely come off as an ad for a bunch of different software. Other than mentioning it here, I do not intend to do anything about that.

Note that I primarily work on OS X and some of the tools mentioned in this post are only available on that platform. You might still find at least inspiration in those areas though – likely very nice Windows and Linux alternatives exist. I did not look.

However if you are not on OS X, but on a tight schedule, just skip all sections other than the “Typing” section (which is probably where you’ll find the best stuff anyway).

Tracking

Everyone has a system for remembering projects and TODOs. If you in stead keep it all in your head, I am glad that you’re not a very busy person – if you are busy, I would worry.

Over time I have tried many different systems for keeping track of it all – having had it as a focus of mine ever since severe physical stress symptoms reminded me that my methods were entirely inadequate.

One tool that I kept coming back to and which I have recently re-focused on is OmniFocus.

While OmniFocus is by no means perfect (I have my fair share of open suggestion posts with the developers), it does provide me with the best feature combination I have seen in this type of tool. Important to me are:

No restrictions or requirements on nesting/indentation depth of items.
- The system has the notion of “tasks” existing in “projects”, but aside from this I can use folder nesting of projects and nesting of tasks as much as I like – assigning whatever meaning I want to that.
Everything is synchronised painlessly between fully-featured clients on my laptop, my phone, and my tablet. And it remains accessible when offline.
Easy, fast, unintrusive task entry.
- The mobile clients have an always accessible “add to inbox” button and the OS X client a global task-entry shortcut.
- The shortcut fires up a little window over whatever you are currently working, allowing you to quickly type in what you need to remember to do, hitting enter when you’re done. Optionally you can add more task details before hitting enter and continuing with whatever you were doing.
  - This is absolutely perfect when you remember a TODO unrelated to your current task or someone comes by your desk with a request.
  - My shortcut for task entry is ctrl+alt+space. I do not recall if this is the default.
- The OS X client also comes with a “service” integration, allowing you to easily create new tasks from other applications – like one based on a file or a snippet of text or perhaps an email.
Optional additional classification and data collection.
- Putting tasks in projects is your absolutely minimum flow. Even that I suppose is optional, as you could simply put all your tasks in your inbox. On top of this, you can scale it as little or as much as makes sense for your particular case. I use everything from no additionals to everything – all dependent on the nature of the task.
  - Aside from putting tasks in optional projects in optional folders, each task can also have an associated context. Different tasks in the same project can have different contexts and contexts can have additional information associated with them – like a location.
  - Assigning a due date to a task will put it in your Forecast view – allowing you to easily view what needs to get done today. This view can also show a quick overview of todays calendar and you can get notifications as the due time of a task is reached.
  - Each task can have an associated attachment. Anything from a snippet of text to a file / an email / whatever.

I use OmniFocus for all project and task tracking. However I do not try to replace my calendar with it. Scheduled events still go in my calendar (I just use iCal) and I also use my calendar to keep track of what I do with my time when on a contract.

This allows me to easily summarize for client reports when needed and I have a little Automator workflow calculate the total number of hours spent contracting this week – allowing me to easily balance the hours.

Typing

Once I am done dreaming up ideas and tasks, like so many others, the majority of my time is spent as a typist – translating those tasks into computer-friendly gibberish of one kind or another. Most of the time I find myself typing C#.

For quite a while I would bounce around between different editors for this task.

VisualStudio with resharper for maximum efficiency – at the cost of either rebooting into Windows or running around in the strange parallel universe of virtualisation.
MonoDevelop for an approximation of the tools provided by VisualStudio, feeling not quite native anywhere, but very beta.
Sublime Text for beautiful, responsive, unintrusive text rendering, with a wonderland of user-configurable keyboard shortcuts and extensions. And absolutely nothing in terms of VS- and MD-like project-wide assistance.

Then a while ago I added a bookmark to the “check this out” pile, for a project called OmniSharp. The promise was simple:

Get full Intellisense-like completion and resharper-like code analysis.
Use it in your text editor of choice.

Ha! Madness.

That was probably at least a half year ago – until I finally found some time to put aside my scepticism for a bit and try to set it up with Sublime.

You know what? It works!

This is what I currently use of the OmniSharp + Sublime Text feature set:

Compile check on save – with errors, warnings, and resharper-like code suggestions.
- Some suggestions I do not agree with and luckily, as with resharper, configuration lets you easily filter those out. Unity also suppresses some warnings when compiling, so I elected to filter those out as well.
File auto-format when triggered by shortcut.
Intellisense-like auto-complete – file, project and library-wide.
Jump to definition / list reference points.
Rename.

I am probably forgetting something, but suffice to say, I no longer feel the need to switch away from Sublime Text to boost efficiency on an unfamiliar codebase.

With regards to shortcuts and configuration, there are a bunch of little tweaks you can set up for Sublime, so of-course I have done exactly that. The good part is that Sublime preferences are all JSON formatted text files, so they are easily shared in a gist:

Preferences in a gist

For text editing, that produces a lovely fullscreen view like the one on the right (click for full size). Tasty, right?

Searching

After lolcat searches and menacingly typing at people who are wrong, I probably spend the majority of my browser time doing reference manual lookups. Extensions like Safari Keyword Search in combination with embedded search engines or Google site search have somewhat streamlined this task, but it never really worked solidly and of-course required that I be online while working.

Enter Dash. I can’t remember who brought it up on twitter, but that’s where I came across it and bookmarked it for later study. The idea is quite simple: It provides an offline search engine for docsets of your choosing – much like man pages.

You get to specify which docsets to download, associate a search key with each docset (IE “cpp:” and “.net:”), and specify a global shortcut for launching the search window – as you would Spotlight on OS X. But wait: There’s more!

Downloadable docsets include topic-specific downloads of wiki-fied Stackoverflow questions and answers. And since you can associate multiple docsets with the same search key, these are easily accessed when you search.

On top of all of this, Dash also comes with an iPad and iPhone client and integration for a bunch of different editors, environments and other tools – including Sublime Text.

You can find my Dash integration settings in the Sublime settings linked above. For global launch shortcut I chose ctrl+shift+space – I do not recall if this is the default.

Navigation

If you, like me, grow a tad too accustomed to the default Sublime Text shortcut of alt+cmd+left/right for tab switching, I would recommend giving it a go as a global shortcut across applications. Configuring System Preferences – Keyboard as shown below did the trick for me. I primarily use it in Terminal and Safari.

And that is about all I have for now. I hope that this is useful to you and do let me know if something in here is off or if you’re more curious about something. If you are at GDC in San Francisco next week, you could even do it in person.

New Gear

Platform

Tracking

Typing

Searching

Navigation

Wrap

Unity, iOS, TeamCity, AppCenter

The [inaudible] X factor

Overview

Unity to iOS

iOS to AppCenter

Build agent setup

Wrap and future

RAID0 NVMe on Ubuntu

Challenge

Process

ext4

Wrapup

A Change of Gears

CoreObject

Challenge

Detour

Implementation

Variants

Unity Protocol Buffers

Overview

Integration

Extraction

Object pooling

Behave 2.7

Contents

Onward

Learn

Basis

Project

Access

Behave 2.6

Contents

Onwards

Trusted Gear

Tracking

Typing

Searching

Navigation