Dev Console Part 5

Dev Console Part 5

Previously Part 4.

The Console is settling and getting to a point where I think I’m ready to split it out into it’s own repository and out of ‘under construction’.

It bothered me that my previous AddStaticTypeByString required you to know the assembly that the thing came from, so instead of using the System.Type version we grab all loaded assemblies and ask them in sequence if they have a type that matches the name.

The big thing I wanted to be able to do is map game data to the console. For me, in Unity, this is typically a ScriptableObject or something a bit fancier. The concept is the same though, there is some object with data that other classes refer to that defines common values. For this I needed to refactor my previous ConsoleHelper Add* to support object instances. Not a big deal, all the Reflection functions in use want an object, we’d just been giving them a null as everything was static previously, now it’s just lots of passing object down. This also meant that if you pass an object you don’t need to pass type and if you pass a type we could assume it was static if we wanted to.

In a more complex terminal window hitting tab repeatedly will cycle through all the possible things you could be trying to type. I have only attempted to get this working from within the App and not via the Browser. To achieve this I’m now keeping the result[] from a call to the complete function and if the autocomplete is requested again before changing the partial phrase we move the index forward through the cached result array and show that text. In the Unity Text we are replacing the text and keeping the current cursor position and setting the cursor end position to the end of the string. There are still some edge cases that are unhandled given my simplistic approach, like moving the cursor manually not resetting the array of autocompletes.

I wanted to be able to find a command without knowing where it might be. This I thought would be especially useful in one of the previously discussed situations where users customise their console but others might want to use it. E.g. you know that you can change gravity, but it isn’t where you thought it would be at UnityEngine.Physics.gravity. The find command then ‘needs’ to be able to traverse every node in the CommandTree but I didn’t want the find command to have to be overly aware of how CommandTree works nor did I want the CommandTree to have a slightly different version of Find that ignores heirarchy and can return more than 1 item. What solved it was a visitor pattern, c# closure (first result curiously not from msdn) and recursion.

We have something like this in CommandTree. Where each command will pass itself to the visitor and if the visitor requests it, then recur for all child commands.

and something like this to use it.

This gets the job done in a way I can deal with, it also meant that I could remove CUDLR’s previous help command, as it built and stored a giant string as commands were added so it could display it later. I didn’t want that giant string to be around always. So now I have an all command, that uses the same visitor style.

In the previous post I mentioned that I profile system might be a useful thing people could create, well the idea stuck so I started adding it. I now store a ScriptableObject with a bunch of csv TextAssets in it.

Example csv

That gets converted to a List of.

And here we are loading that example csv that is called Default in the console to add additional elements.

One other bit I found I needed was an attribute to flag, fields, props, methods or classes to be ignored by a AddAllToConsole. I created an attribute called CommandIgnore that I can apply to any thing and then my console helper variants will skip over it. The primary use I have for this is being able to have static classes or global data that I want in the console but that still allows me to refactor those classes in the usual way and not worry about exposing partial or useless functions to the console.

Dev Console Part 4

Dev Console Part 4

Previously Part 3.

In the previous post I showed CUDLR from within Unity. This is done by some simple uGUI and a script that passes a line of text to CUDLR.Console.Run when the appropriate key is hit. I then added a callback to CUDLR to so I could get notifed when it wanted to log things so I could show them in my GUI elements. I didn’t want to just pipe these through Debug.Log as they are used for the console communicating with it’s user and are not things that need to show up in the Debug.Log peeking I have set up or in a file log if someone wanted to save all Debug.Logs out.

I also made a handful of small changes to CUDLR;

  • It stored items in its CommandTree via CommandAttributes directly, this irked me when it manually made one that wasn’t actually attached to anything. This means some data is duplicated now but nothing reports to be an attribute when it actually isn’t
  • used spaces as both the separator between child elements and parameters. I switched this to the more familiar ‘.’ for child elements.
    • This also meant that I refactored the few places it was previously relying on being able to regex or split on ‘ ‘ and made a dedicated function for extracting command location from parameters.
  • CommandTree had several recursive functions that did similar things for finding the desired command, then running it or auto completing name or adding a new command. I refactored that into a more general FindCommand function, this was easier and required because I also wanted to switch from spaces to ‘.’ between child objects.
  • It’s function signature took string[]. Simplified that to string as I wanted my auto gen wrappers to do the conversion for me between a parameter string to object[].
  • Command entries in the tree didn’t know their name, only the parent object did. Added field to each entry so it knows what it is.
  • This allowed me to change how commands are stored to always be lower case and their own name storage to keep original cases. Which meant that I could easily make the auto completion case insensitive.
  • Made CUDLR.Console automatic registration of commands use the RegisterCommand path instead of slightly different method for them, means less things to test as now it is just a big loop that then uses the same thing everything else does.
  • Made RegisterCommand use the auto generated wrapper we built up in previous posts, this removed the need for CUDLR’s two delegates one with no params one with string[]

That’s great but lets talk autocomplete. The previous default behavior in the browser is you hit tab, and it uses the Complete method which returns an array of all partial matches. If there is only 1 partial match then replace the input box with that text. I wanted a bit more, if I type ‘ti’ and the only partial match is ‘Time’ then I run another auto complete on Time so it gives me a list of all the sub commands within the holder.

Similarly, if I type ‘phy’ but I have ‘Physics’ and ‘Physics2D’, I don’t want to have to type the ‘sics’. It should be smart enough to complete that part for me. I use something similar to this to determine it.

I found some fun consequences of being able to automatically map objects and statics to the console, much of CUDLR.Console is static, so we can add the console to the console. Which means that we can run console commands via the console. Right now not overly useful, but I really enjoy that it just works and it may become useful if we push the console further towards a way of configuring and creating behaviours.

I was also able to get the console to add new types to itself at runtime. I enter this into the console.

Which ends up calling code that is something like this.

This is crazy, we could now remotely connect to a running game via the browser, make it expose static data to us (as long as we know the full class name and the assembly it comes from). Without us even considering that we might have wanted to do that at build time. This also means that if ports are open you could play some amazing pranks on your colleagues. More practically it means you could have different sets of console commands exposed by default to different team members, eg artist wants lots of graphics and light settings exposed by don’t want to wade through physics or game balance data. But the moment they want access to other things they can add that to their console. It would might help in, “Um, can you come look at this weird thing” type situations, having another team member come over and add the elements to the console that they are used to and dig in to find out what is happening, they could also do this remotely, just post your IP in a slack channel.

In the future I’ll be looking into non-static things, most likely with a focus on data in ScriptableObjects.

Part 5 here.

Dev Console Part 3

Dev Console Part 3

Previously Part 2.

Now to get to making binding easy to do. GConsole, CUDLR, etc. all make use of c# Attributes to attach additional information to methods. If you’ve used Unity much you probably have seen Range or Tooltip, these are used by unity editor drawer code to change how things are shown in the inspector. The existing console systems use attributes to give names, descriptions, help text and other data to methods that the console system can get access to and report to the user, CUDLR attribute here. Then the console system can find all types that have these attributes and add them to their internal list of commands. So that’s understood and kinda solved.

I wanted to be able to add methods, fields and properties to the console without having to attach attributes to them. Partly, if it isn’t stupid easy to do, I know at some point I’ll stop adding things to the console as a matter of course. The other part is that I want to be able to add elements to the console that I cannot add attributes to, for example Unity Physics class. A solution involving writing a facade for Unity built-ins just so I can add attributes to them isn’t a good use of anyone’s time. Here’s what I used while determining the method.

Reflection to the rescue again. AddAllStaticsToConsole uses GetMethods with BindFlags to filter non public and non static items out, we also then fitler out all ‘special’ methods, these things that get auto generated for properties, the get_ and set_. It also uses GetFields and GetProperties in a similar fashion. These all have their own types but basically we loop through the array of results and try to add them to the console if they take are convertible (we discussed this in Part 2) and we know the name from the reflection information. We refactor this into related sub functions so we can reuse them as this gets more powerful.

Moment of truth.

At this point, I’ve also switched to CUDLR from GConsole. CUDLR it’s built to work remotely via the browser and it made more sense add in-game access to CUDLR rather than make GConsole usable from the browser. CUDLR also already had the concept of a hierarchy which I wanted. I ended up making a number of changes and adding some features to CUDLR which I’ll go into in the next part.

Part 4 here.

Dev Console Part 2

Part 1 here.

Working with GConsole and want to be able to wrap existing functions more automatically

 

That’s how we get things in the console. In GConsole then when you eval the input string it matches the first part against all ‘command’ that have been registered. If it finds a match it calls the ‘method’ with the substring of the input string from after the command name till end of string. So for ease of use I wanted to be able to wrap an arbitrary function in a Func<string,string>.

That’s a few separate tasks

  • Converting string to type
  • Determining number of parameters
  • Knowing param types in the first place

Determining number of params. To start with during testing I simply used .Splint(“,”) on the inputstring, this meant that as long they were comma separated they where different. I knew this was just to get other things tested out as things like sentences within quotes and vector3s etc. want commas in them but not to be a separate param. Regex was the solution as things got more complicated.

 

 

This will hold out for a while but it’s not going to handle object hierarchies, like json, should the Console get to that level of complexity in the future.

 

Converting strings to types, assuming we know what it should be. ints and floats etc. have builtin Parse functions, if it’s a string, it already is one. For the immediate future, vectors were the other thing I really wanted, they just require breaking up the string and using multiple float Parse. Now we need a way to access these in a way that makes sense for our use case. For me it was a lookup, keyed by Type and storing a Func<object, string>. When these are eventually used in an automated fashion we put the conversion inside try catch blocks so if the conversion fails we report that to the user via the console output and don’t call the function.

 

Knowing parameter types in the first place. Reflection is very powerful. In fact we’ll end up using it for far more than just this. We get the method info out of the type. That can give us attributes applied to the method, which GConsole and CUDLR etc. already use to mark up classes or methods with names that they should be stored under in the console as well as description text to display to the user. What I need right now though is the the parameters of the method. Then we get something like this.

 

 

Next step will be more automating binding and now that we’ve proved out some concepts, moving to CUDLR.

Part 3 here.

Dev Console Part 1

Dev Console Part 1

So I was working on a project that targets mobile and I’d run into a problem that didn’t replicate in editor or on Android, only on iOS. I really just wanted to put a bunch of logs in and see what was happening. I wanted an in game dev console. If you aren’t sure what I mean by that, if you’ve played with an Id or Valve game you’ve probably enabled developer console at some point, seen logs pop up in the top left or hit ` to change maps or enable net_graph.

Similarly, Bethesda games tend to have powerful in game consoles that can execute the same sort of commands as the scripting language. Which leads to amazing things like this Monster Factory Fallout4.

So I went hunting, surely this is something people have already attempted, if not solved.

https://github.com/mminer/consolation
https://github.com/Wenzil/UnityConsole
https://github.com/gzuidhof/GConsole
https://github.com/hecomi/uREPL
https://github.com/proletariatgames/CUDLR
https://github.com/SpaceMadness/lunar-unity-console
Yep, people have tried this before.

To fix my problem I just needed to see logs so I went with something very similar to Consolation. It uses the old immediate mode Unity GUI calls. I didn’t care for that, pixel based sizes and GUI Skins are not fun to work with. The idea is the same though, hook up Unity Log Callback and append some strings to an auto scrolling UI.Text. I also made it fade out nicely with a LeanTween.

It did let me find my problem, it was a file and directory formatting problem that meant a file was failing to be found on iOS. Fixed.

I left the dev console enabled and found that I like having the logs visible during development. It makes me feel more aware of how the game is running and more connected to the code that I wrote. Rather than it feeling like a disconnected artifact of a build process. For this particular project I had already started building a developer only panel, that gives run-time access to a number of slider values and player resets and unlock shortcuts. So the logical step was to make this an actual console not just a way of seeing logs and move the functionality that I was manually building and adding to the . For this I looked to GConsole as it was simple to integrate and small enough that I could read all the code myself if it didn’t meet my needs. I quickly found that I wanted more from it.

I want to keep working on this and add more features to it, no one of the consoles I found that already exists quite does all the things that I want.

So here are some non-ordered requirements;

  • See logs
    • preview as they come in
    • expand console to scroll back through them
  • Run commands, eg quit app, load a scene, give player xp
  • Inspect and change vars, eg player’s hp, gravity, spawn timer
  • Inspect and change cl vars, eg resolution, vsync
  • Get snapshots of data, eg. GameObject hierarchy, Active enemies
  • Not have to reinvent all functionality that exists elsewhere just for the console
    • This is crucial, if functionality that already has to be manually wrapped or duplicated for the console to access it, then it just doesn’t get done as the cost of get that functionality wrapped and configurable by the console vs adding new features or bug fixing or polishing just doesn’t hold up.

Some examples of what we would like the console input and outputs to look like

GConsole has a simple to understand code base, part of how it achieves this is by only accepting Func<string, string> as methods. This pushes the developer to wrap existing functionality in a function that takes a single string that could be anything from the user. I know that I’d get sick of writing string extraction and conversion code and wrapping existing functionality.

CUDLR is awesome, it operates like a slim http server in your game that you write custom routes (routes a common web concept). It’s designed to be able to remotely debug and see the console via the browser, so locally or if you know the IP you can get the console logs and send commands and fetch routes. But it ships with no built in way of doing that from within the game itself which I’d want to do when I’m on a phone or tablet. CUDLR still gives a string[] as the parameter to it’s commands and would require custom wrapping of existing functions.

I’m going to keep experimenting and pushing this forward. Should you wish to see some in progress code you can check out https://bitbucket.org/steve_halliwell/aid-common console stuff is currently in UnderConstruction/ConsoleHelper

Part 2 here.