Nov
4
2016

Using Accord.NET with Unity

An article by mgear

accord-net-unity-gaussian

While looking for Harris Corner Detection examples, without using openCV, founded this framework:
“The Accord.NET Framework is a machine learning framework combined with audio and image processing libraries completely written in C#”

Website: http://accord-framework.net
Github: https://github.com/accord-net/framework

*Note: There is updated tutorial here with images: https://github.com/accord-net/framework/wiki/Unity

How to get it running with Unity
– Open new empty project in Unity (I Used 5.5.0b10)
– Edit/Project Settings/Player, set Api compatibility Level to .NET2.0 (instead of .NET2.0 Subset)
– Download framework from https://github.com/accord-net/framework/releases ( I took v3.3.0 )
– Copy DLL’s from that package (from “Release/net35/” folder) into your projects Assets/Plugins/ folder
– Also copy System.Drawing.dll from your c: drive into “Assets/Plugins/” folder (should be version 2.0, I founded mine from “C:\Windows\Microsoft.NET\Framework\v2.0.50727\”)
– Create c# script and use the code below (*This was just a quick test to get it running, testing Gaussian filter on texture)
– See my scene setup screenshot below the source

	using System.Collections;
	using UnityEngine;
	using Accord.Imaging.Filters;

	// More info: http://unitycoder.com/blog/2016/11/04/using-accord-net-with-unity/

	public class AccordFrameworkTest : MonoBehaviour
	{
	public Renderer sourceRenderer; // our source texture is assigned on this gameobject
	public Renderer targetRenderer;

	void Start()
	{
	// get maintexture from source renderer
	var sourceTexture = (Texture2D)sourceRenderer.material.mainTexture;

	// create system bitmap
	System.Drawing.Bitmap accordImage = new System.Drawing.Bitmap(sourceTexture.width, sourceTexture.height);

	// copy our texture pixels to that system bitmap
	for (int x = 0; x < accordImage.Width; x++)
	{
	for (int y = 0; y < accordImage.Height; y++)
	{
	var c = (Color32)sourceTexture.GetPixel(x, y);
	var nc = System.Drawing.Color.FromArgb(c.r, c.g, c.b, c.a);
	accordImage.SetPixel(x, y, nc);
	}
	}


	// These 2 lines are the only Accord.NET code used here, create filter and then apply it to bitmap
	IFilter gaussianFilter = new GaussianBlur(2.0, 20);
	var resultsAccordImage = gaussianFilter.Apply(accordImage);


	// copy result pixels into our color32 array from system.drawing.bitmap
	var colors = new Color32[resultsAccordImage.Width * resultsAccordImage.Height];
	for (int x = 0; x < resultsAccordImage.Width; x++)
	{
	for (int y = 0; y < resultsAccordImage.Height; y++)
	{
	var c = resultsAccordImage.GetPixel(x, y);
	colors[y * resultsAccordImage.Width + x] = new Color32(c.A,c.R, c.G, c.B); // colors are flipped in System.Drawing.Color
	}
	}

	// create new results texture from that color32 array
	var targetTexture = new Texture2D(sourceTexture.width, sourceTexture.height, sourceTexture.format, false);
	targetTexture.SetPixels32(colors);
	targetTexture.Apply(false);

	// assign it to another object
	targetRenderer.material.mainTexture = targetTexture;
	}
	}

view raw

AccordFrameworkTest.cs

hosted with ❤ by GitHub

Image#1: Scene setup screenshot
– Add 2 Quads with different materials (use Unlit/Texture shader)
– Assign your source texture into QuadSource *Note: Texture importer settings must have [x] Read/Write enabled
– Assign those 2 quads into the script (which you have in some gameobject in scene) and hit Play!

accord-net-unity-tutorial

*Main image: Using Gaussian filter from Accord.NET

Tags: accord.net, detection, dll, framework, harris feature, machine learning

About the Author:

.fi

12 Comments + Add Comment

cvnk 05/11/2016 at 05:35

Thanks for bringing this to my attention. I don’t need it for Unity but we do occasionally use OpenCV with at my job and it can be a struggle using it with .NET (even with the popular wrappers available). This looks like an interesting potential alternative.

Reply
C#で使える機械学習ライブラリ『Accord.NET Framework』 10/01/2017 at 22:03

[…] 少し調べたら、Unity5.5でAccord.NETを動作させるチュートリアルがあった。↓ Using Accord.NET with UnityWhile looking for Harris Corner Detection examples, without using openCV, founded this framework: […]

Reply
mgear 10/03/2017 at 21:57

Getting Started With OpenCvSharp 3
https://www.codeproject.com/tips/1085960/getting-started-with-opencvsharp

Reply
mgear 12/05/2017 at 23:43

“Anyone know a machine learning library for C sharp?”
https://forum.unity3d.com/threads/anyone-know-a-machine-learning-library-for-c-sharp.470614/

http://numl.net/
http://www.heatonresearch.com/encog/

Reply
Wendy 21/07/2017 at 18:53

Hello, When I followed the above steps install the Accord.net, but it has one problem like the following:
TypeLoadException: Could not load type ‘Accord.DirectSound.AudioDeviceInfo’ from assembly ‘Accord.Audio.DirectSound, Version=3.3.0.0, Culture=neutral.

So May I ask how to solve this problem? Thanks

Reply
- mgear 21/07/2017 at 20:31
  
  seems to happen in 2017.x, but not in that 5.5.x.. would be interesting to know what changed.
  
  Reply
  - Wendy 21/07/2017 at 20:53
    
    But I tried 5.5.0f3 at first, it still has similar problem. I used accord. Net at version 3.3, and follow your above steps. So would you please help me what I should fix? Thank you so much!
    
    Reply
    - mgear 21/07/2017 at 22:06
      
      in 5.5.3p4 it gives notification message (not error like in 2017) “SpritePacker failed to get types from Accord.Audio.DirectSound, Version=3.3.0.0, Culture=neutral, PublicKeyToken=fa1a88e29555ccf7. Error: The classes in the module cannot be loaded.
      UnityEditor.Sprites.Packer:GetSelectedPolicyId()”
      
      but only if spritepacker is enabled.. could always delete that directsound dll for now
      
      Reply
      - Brad Knox 23/08/2017 at 08:29
        
        The SpritePacker error is an unhelpful error that indicates that Assembly.GetTypes() failed, which means that some dependency of Accord.Audio.DirectSound is missing.
        
        What dependency? The error doesn’t say. But I resolved a similar issue when I tried to build the project for iOS, at which point I got an error that told me exactly what was missing.
Brad Knox 23/08/2017 at 09:16

I got this working on macOS after 10+ hours of trying. Here’s my documentation of what I did: https://github.com/accord-net/framework/issues/799#issuecomment-324157507.

Curiously, unless some of the items I mention in the bottom bullets were necessary, it turned out to be the steps from this post, but without needing System.Drawing.dll. I originally got thrown off because I couldn’t find System.Drawing.dll on my computer, but that link above also points to where I eventually found it (though I didn’t keep it in Assets).

Reply
mgear 23/09/2017 at 11:33

there is some updated tutorial here,
Installing the Accord.NET Framework in Unity
https://github.com/accord-net/framework/wiki/Unity

Reply
Cody 13/12/2022 at 13:45

U really expect me to code to run Accord.Net ? Madman’s developers developing Libraries to be used only by developers. No install packages. Why don’t u say so loud and clear. Why i have to install oversize’d software packages in order to sun few lines of code mainly CLI driven with commands that needs to be learn first ? Why i have to code it in order to be able to run it ? If this approach would be used from Microsoft then no user will use it. That’s why such packages and Libraries are widely learned and used !? U cant even start to learn about ANN’s what so ever if u cant run it ? Right ?

Reply