Unit testing is a valuable tool that a lot of studios fail to implement. Unit testing can save a great amount of time and energy (and therefore money) by allowing developers, tool writers, and even content creators to catch broken deliverables before passing them downstream in the pipeline. The lack of unit testing with TD's is most likely due to one of the following:
- The person doesn't know what unit testing is.
- The person has heard of unit testing but has no idea how to implement it.
- The person is in production and believes there is no time for creating unit tests.
- The person knows what unit tests are and just doesn't want to do them.
In this post, I am going to describe unit testing as it relates to assets and Python tools created for Maya. For those that don't know, unit testing is pretty much a bunch of Python functions that you write that verify your tool is doing what it is supposed to be doing. For example, if you write an ik/fk switch tool, one unit test could be to load a rig, pose the arm in ik, switch to fk, and then check to see if the arm is in the same place. Another example is if you write a blendShape exporter, you can write tests to make sure the shapes are properly recreated when you import them into a new scene. Unit testing gives developers the confidence to update their code and assets with the certainty that their updates are not breaking any existing functionality.
If you are writing any semi-complex tool that is being used in production by other users, it is extremely worth while to write unit tests for your tool. Not only will it make it easier for other developers to contribute to your work, it also reduces the amount of regressions, or updates that break other parts of your tool, from happening. In today's bigger studios, you will often be tasked with updating or maintaining systems that you did not write. If there are no testing suites created for the tool, you have no way of knowing if the updates you do will break some part of the system. If I were in the heat of production, I would rather spend a few hours over a day or two writing tests for a tool, then spending days over an entire production constantly chasing down products of buggy tools.
I'm not going to go over the intricacies of using Python's unittest package as there are already many resources describing how to use unittest. I will however describe it's relationship with Maya and how you can use it inside of Maya.
All of the code described in this post is available on my Github.
To facilitate writing tests for Maya, it is useful to implement a subclass of unittest.TestCase. The derived class will add helpful functionality such as automatic loading/unloading of plug-ins, and deleting temporary files:
Derived TestCase
class TestCase(unittest.TestCase):
"""Base class for unit test cases run in Maya.
Tests do not have to inherit from this TestCase but this derived TestCase contains convenience
functions to load/unload plug-ins and clean up temporary files.
"""
# Keep track of all temporary files that were created so they can be cleaned up after
# all tests have been run
files_created = []
# Keep track of which plugins were loaded so we can unload them after all tests have been run
plugins_loaded = set()
@classmethod
def tearDownClass(cls):
super(TestCase, cls).tearDownClass()
cls.delete_temp_files()
cls.unload_plugins()
@classmethod
def load_plugin(cls, plugin):
"""Load the given plug-in and saves it to be unloaded when the TestCase is finished.
@param plugin: Plug-in name.
"""
cmds.loadPlugin(plugin, qt=True)
cls.plugins_loaded.add(plugin)
@classmethod
def unload_plugins(cls):
# Unload any plugins that this test case loaded
for plugin in cls.plugins_loaded:
cmds.unloadPlugin(plugin)
cls.plugins_loaded = []
@classmethod
def delete_temp_files(cls):
"""Delete the temp files in the cache and clear the cache."""
# If we don't want to keep temp files around for debugging purposes, delete them when
# all tests in this TestCase have been run
if Settings.delete_files:
for f in cls.files_created:
if os.path.exists(f):
os.remove(f)
cls.files_create = []
@classmethod
def get_temp_filename(cls, file_name):
"""Get a unique filepath name in the testing directory.
The file will not be created, that is up to the caller. This file will be deleted when
the tests are finished.
@param file_name: A partial path ex: 'directory/somefile.txt'
@return The full path to the temporary file.
"""
temp_dir = Settings.temp_dir
if not os.path.exists(temp_dir):
os.makedirs(temp_dir)
base_name, ext = os.path.splitext(file_name)
path = '{0}/{1}{2}'.format(temp_dir, base_name, ext)
count = 0
while os.path.exists(path):
# If the file already exists, add an incrememted number
count += 1
path = '{0}/{1}{2}{3}'.format(temp_dir, base_name, count, ext)
cls.files_created.append(path)
return path
With these added utility methods, you can write tests that verify that your plug-ins can be unloaded properly and that any temporary files are cleaned up:
Sample Test using Derived TestCase
import maya.cmds as cmds
from cmt.test import TestCase
class SampleTests(TestCase):
@classmethod
def setUpClass(cls):
cls.load_plugin('myBlendShapeExportingPlugin')
def test_export_blendshape(self):
base = cmds.polySphere(r=5)[0]
target = cmds.polySphere(r=10)[0]
blendshape = cmds.blendShape(target, base)[0]
file_path = self.get_temp_filename('blendshape.shapes')
cmds.myCustomBlendShapeExporter(blendshape, path=file_path)
self.assertTrue(os.path.exists(file_path))
Now that we can write tests, I'll go over how to run them from outside of Maya from the commandline without opening the Maya interface. This is useful to quickly run tests without having to open up the Maya application.
Running Tests from Commandline
def main():
parser = argparse.ArgumentParser(description='Runs unit tests for a Maya module')
parser.add_argument('-m', '--maya',
help='Maya version',
type=int,
default=2016)
pargs = parser.parse_args()
mayaunittest = os.path.join(CMT_ROOT_DIR, 'scripts', 'cmt', 'test', 'mayaunittest.py')
cmd = [mayapy(pargs.maya), mayaunittest]
if not os.path.exists(cmd[0]):
raise RuntimeError('Maya {0} is not installed on this system.'.format(pargs.maya))
maya_app_dir = create_clean_maya_app_dir()
# Create clean prefs
os.environ['MAYA_APP_DIR'] = maya_app_dir
# Clear out any MAYA_SCRIPT_PATH value so we know we're in a clean env.
os.environ['MAYA_SCRIPT_PATH'] = ''
# Run the tests in this module.
os.environ['MAYA_MODULE_PATH'] = CMT_ROOT_DIR
try:
subprocess.check_call(cmd)
except subprocess.CalledProcessError:
pass
finally:
shutil.rmtree(maya_app_dir)
if __name__ == '__main__':
main()
The above code is the entry point to a Python script that runs the tests from the command line. You'll notice that in order for the tests to be run, we need to use the mayapy Python interpreter that ships with Maya. The above script will usually be executed using the system install of Python and then a subprocess will be spawned using the mayapy of the specified Maya version:
Getting the mayapy Path
def get_maya_location(maya_version):
"""Get the location where Maya is installed.
@param maya_version The maya version number.
@return The path to where Maya is installed.
"""
if 'MAYA_LOCATION' in os.environ.keys():
return os.environ['MAYA_LOCATION']
if platform.system() == 'Windows':
return 'C:/Program Files/Autodesk/Maya{0}'.format(maya_version)
elif platform.system() == 'Darwin':
return '/Applications/Autodesk/maya{0}/Maya.app/Contents'.format(maya_version)
else:
location = '/usr/autodesk/maya{0}'.format(maya_version)
if maya_version < 2016:
# Starting Maya 2016, the default install directory name changed.
location += '-x64'
return location
def mayapy(maya_version):
"""Get the mayapy executable path.
@param maya_version The maya version number.
@return: The mayapy executable path.
"""
python_exe = '{0}/bin/mayapy'.format(get_maya_location(maya_version))
if platform.system() == 'Windows':
python_exe += '.exe'
return python_exe
Also, we want to run the tests in a predictable and clean environment. It is very hard to track down bugs when external environment changes can affect your tools. Therefore, we will be creating a fresh directory that contains a clean Maya.env and Maya preferences which is referred to by the MAYA_APP_DIR environment variable:
Creating a Clean MAYA_APP_DIR
def create_clean_maya_app_dir():
"""Creates a copy of the clean Maya preferences so we can create predictable results.
@return: The path to the clean MAYA_APP_DIR folder.
"""
app_dir = os.path.join(CMT_ROOT_DIR, 'tests', 'clean_maya_app_dir')
temp_dir = tempfile.gettempdir()
if not os.path.exists(temp_dir):
os.makedirs(temp_dir)
dst = os.path.join(temp_dir, 'maya_app_dir{0}'.format(uuid.uuid4()))
shutil.copytree(app_dir, dst)
return dst
Before running the mayapy subprocess, we set the three Maya environment variables MAYA_APP_DIR, MAYA_SCRIPT_PATH, and MAYA_MODULE_PATH. The MAYA_APP_DIR is set to the clean preferences directory described above. The MAYA_SCRIPT_PATH is cleared just in case there is an existing value set. Current values of MAYA_SCRIPT_PATH could run userSetup files located in other places that we do not want in our testing environment. The MAYA_MODULE_PATH is currently set to the module that this code is located in (available on my Github). However, this could be updated to load all the modules stored at some location in your development environment or shared network location.
The mayapy subprocess gets spawned and calls the following function:
Intialize maya.standalone.
def run_tests_from_commandline():
"""Runs the tests in Maya standalone mode.
This is called when running cmt/bin/runmayatests.py from the commandline.
"""
import maya.standalone
maya.standalone.initialize()
# Make sure all paths in PYTHONPATH are also in sys.path
# When a maya module is loaded, the scripts folder is added to PYTHONPATH, but it doesn't seem
# to be added to sys.path. So we are unable to import any of the python files that are in the
# module/scripts folder. To workaround this, we simply add the paths to sys ourselves.
realsyspath = [os.path.realpath(p) for p in sys.path]
pythonpath = os.environ.get('PYTHONPATH', '')
for p in pythonpath.split(os.pathsep):
p = os.path.realpath(p) # Make sure symbolic links are resolved
if p not in realsyspath:
sys.path.insert(0, p)
run_tests()
# Starting Maya 2016, we have to call uninitialize
if float(cmds.about(v=True)) >= 2016.0:
maya.standalone.uninitialize()
To run the tests, we'll first initialize maya.standalone. Then there seems to be buggy behavior that we'll account for where scripts directories of Maya modules do not properly get added to the sys.path even though they get added to the PYTHONPATH. Once Maya is initialized and ready to go we can run the tests:
Run the Tests
def run_tests(directories=None, test=None, test_suite=None):
"""Run all the tests in the given paths.
@param directories: A generator or list of paths containing tests to run.
@param test: Optional name of a specific test to run.
@param test_suite: Optional TestSuite to run. If omitted, a TestSuite will be generated.
"""
if test_suite is None:
test_suite = get_tests(directories, test)
runner = unittest.TextTestRunner(verbosity=2, resultclass=TestResult)
runner.failfast = False
runner.buffer = Settings.buffer_output
runner.run(test_suite)
Before we can run the tests, we first have to find them. Finding the tests can be tricky if there is no agreed upon standard. I will assume all tests will be located in a “tests” directory of each Maya module in the MAYA_MODULE_PATH:
Finding the Tests
def get_tests(directories=None, test=None, test_suite=None):
"""Get a unittest.TestSuite containing all the desired tests.
@param directories: Optional list of directories with which to search for tests. If omitted, use all "tests"
directories of the modules found in the MAYA_MODULE_PATH.
@param test: Optional test path to find a specific test such as 'test_mytest.SomeTestCase.test_function'.
@param test_suite: Optional unittest.TestSuite to add the discovered tests to. If omitted a new TestSuite will be
created.
@return: The populated TestSuite.
"""
if directories is None:
directories = maya_module_tests()
# Populate a TestSuite with all the tests
if test_suite is None:
test_suite = unittest.TestSuite()
if test:
# Find the specified test to run
directories_added_to_path = [p for p in directories if add_to_path(p)]
discovered_suite = unittest.TestLoader().loadTestsFromName(test)
if discovered_suite.countTestCases():
test_suite.addTests(discovered_suite)
else:
# Find all tests to run
directories_added_to_path = []
for p in directories:
discovered_suite = unittest.TestLoader().discover(p)
if discovered_suite.countTestCases():
test_suite.addTests(discovered_suite)
# Remove the added paths.
for path in directories_added_to_path:
sys.path.remove(path)
return test_suite
def maya_module_tests():
"""Generator function to iterate over all the Maya module tests directories."""
for path in os.environ['MAYA_MODULE_PATH'].split(os.pathsep):
p = '{0}/tests'.format(path)
if os.path.exists(p):
yield p
The above code populates a unittest.TestSuite object with all the tests found in the MAYA_MODULE_PATH directories. There is also additional parameters to just load individual tests which we will revisit in a future post.
In the run_tests function above, we pass in a custom result class to the TextTestRunner constructor. The custom TestResult class provides functionality such as disabling logging during the test run as well as suppressing script editor output. Often it is useful to just see the test results without all the script editor clutter. The custom TestResult class also creates a new scene after each test is executed. When writing tests, you want each test to be totally independent from each other. Creating a new scene file after each test helps ensure the tests are independent.
Custom TestResult
class TestResult(unittest.TextTestResult):
"""Customize the test result so we can do things like do a file new between each test and suppress script
editor output.
"""
def startTestRun(self):
"""Called before any tests are run."""
super(TestResult, self).startTestRun()
ScriptEditorState.suppress_output()
if Settings.buffer_output:
# Disable any logging while running tests. By disabling critical, we are disabling logging
# at all levels below critical as well
logging.disable(logging.CRITICAL)
def stopTestRun(self):
"""Called after all tests are run."""
if Settings.buffer_output:
# Restore logging state
logging.disable(logging.NOTSET)
ScriptEditorState.restore_output()
if Settings.delete_files and os.path.exists(Settings.temp_dir):
shutil.rmtree(Settings.temp_dir)
super(TestResult, self).stopTestRun()
def stopTest(self, test):
"""Called after an individual test is run.
@param test: TestCase that just ran."""
super(TestResult, self).stopTest(test)
if Settings.file_new:
cmds.file(f=True, new=True)
With all that code in place, you should be able to run your tests from the command line and see which tests pass and which tests fail:
D:devmayacmtbin>python runmayatests.py --maya 2016
test_another_thing (test_simple.SimpleTests) ... FAIL
test_error (test_simple.SimpleTests) ... ok
test_skip (test_simple.SimpleTests) ... skipped 'Skip this for now'
test_something (test_simple.SimpleTests) ... ok
======================================================================
FAIL: test_another_thing (test_simple.SimpleTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "D:devmayacmtteststest_simple.py", line 13, in test_another_thing
self.assertTrue(False)
AssertionError: False is not true
----------------------------------------------------------------------
Ran 4 tests in 0.031s
FAILED (failures=1, skipped=1)
All of this code is available to look at in my new cmt (Chad's Maya Tools…I know, not very creative) repository on Github. Also, I would like to thank my former colleague, Sivanny, for providing some of the ideas described in this post. In a future post, I will go over the creation of a user interface to run tests interactively inside of Maya.