Nov
4
2016

Using Accord.NET with Unity

accord-net-unity-gaussian

While looking for Harris Corner Detection examples, without using openCV, founded this framework:
“The Accord.NET Framework is a machine learning framework combined with audio and image processing libraries completely written in C#”

Website: http://accord-framework.net
Github: https://github.com/accord-net/framework

*Note: There is updated tutorial here with images: https://github.com/accord-net/framework/wiki/Unity

How to get it running with Unity
– Open new empty project in Unity (I Used 5.5.0b10)
– Edit/Project Settings/Player, set Api compatibility Level to .NET2.0 (instead of .NET2.0 Subset)
– Download framework from https://github.com/accord-net/framework/releases ( I took v3.3.0 )
– Copy DLL’s from that package (from “Release/net35/” folder) into your projects Assets/Plugins/ folder
– Also copy System.Drawing.dll from your c: drive into “Assets/Plugins/” folder (should be version 2.0, I founded mine from “C:\Windows\Microsoft.NET\Framework\v2.0.50727\”)
– Create c# script and use the code below (*This was just a quick test to get it running, testing Gaussian filter on texture)
– See my scene setup screenshot below the source


using System.Collections;
using UnityEngine;
using Accord.Imaging.Filters;
// More info: http://unitycoder.com/blog/2016/11/04/using-accord-net-with-unity/
public class AccordFrameworkTest : MonoBehaviour
{
public Renderer sourceRenderer; // our source texture is assigned on this gameobject
public Renderer targetRenderer;
void Start()
{
// get maintexture from source renderer
var sourceTexture = (Texture2D)sourceRenderer.material.mainTexture;
// create system bitmap
System.Drawing.Bitmap accordImage = new System.Drawing.Bitmap(sourceTexture.width, sourceTexture.height);
// copy our texture pixels to that system bitmap
for (int x = 0; x < accordImage.Width; x++)
{
for (int y = 0; y < accordImage.Height; y++)
{
var c = (Color32)sourceTexture.GetPixel(x, y);
var nc = System.Drawing.Color.FromArgb(c.r, c.g, c.b, c.a);
accordImage.SetPixel(x, y, nc);
}
}
// These 2 lines are the only Accord.NET code used here, create filter and then apply it to bitmap
IFilter gaussianFilter = new GaussianBlur(2.0, 20);
var resultsAccordImage = gaussianFilter.Apply(accordImage);
// copy result pixels into our color32 array from system.drawing.bitmap
var colors = new Color32[resultsAccordImage.Width * resultsAccordImage.Height];
for (int x = 0; x < resultsAccordImage.Width; x++)
{
for (int y = 0; y < resultsAccordImage.Height; y++)
{
var c = resultsAccordImage.GetPixel(x, y);
colors[y * resultsAccordImage.Width + x] = new Color32(c.A,c.R, c.G, c.B); // colors are flipped in System.Drawing.Color
}
}
// create new results texture from that color32 array
var targetTexture = new Texture2D(sourceTexture.width, sourceTexture.height, sourceTexture.format, false);
targetTexture.SetPixels32(colors);
targetTexture.Apply(false);
// assign it to another object
targetRenderer.material.mainTexture = targetTexture;
}
}

Image#1: Scene setup screenshot
– Add 2 Quads with different materials (use Unlit/Texture shader)
– Assign your source texture into QuadSource *Note: Texture importer settings must have [x] Read/Write enabled
– Assign those 2 quads into the script (which you have in some gameobject in scene) and hit Play!

accord-net-unity-tutorial

 

*Main image: Using Gaussian filter from Accord.NET


12 Comments + Add Comment

  • Thanks for bringing this to my attention. I don’t need it for Unity but we do occasionally use OpenCV with at my job and it can be a struggle using it with .NET (even with the popular wrappers available). This looks like an interesting potential alternative.

  • […] 少し調べたら、Unity5.5でAccord.NETを動作させるチュートリアルがあった。↓ Using Accord.NET with UnityWhile looking for Harris Corner Detection examples, without using openCV, founded this framework: […]

  • Hello, When I followed the above steps install the Accord.net, but it has one problem like the following:
    TypeLoadException: Could not load type ‘Accord.DirectSound.AudioDeviceInfo’ from assembly ‘Accord.Audio.DirectSound, Version=3.3.0.0, Culture=neutral.

    So May I ask how to solve this problem? Thanks

    • seems to happen in 2017.x, but not in that 5.5.x.. would be interesting to know what changed.

      • But I tried 5.5.0f3 at first, it still has similar problem. I used accord. Net at version 3.3, and follow your above steps. So would you please help me what I should fix? Thank you so much!

        • in 5.5.3p4 it gives notification message (not error like in 2017) “SpritePacker failed to get types from Accord.Audio.DirectSound, Version=3.3.0.0, Culture=neutral, PublicKeyToken=fa1a88e29555ccf7. Error: The classes in the module cannot be loaded.
          UnityEditor.Sprites.Packer:GetSelectedPolicyId()”

          but only if spritepacker is enabled.. could always delete that directsound dll for now

          • The SpritePacker error is an unhelpful error that indicates that Assembly.GetTypes() failed, which means that some dependency of Accord.Audio.DirectSound is missing.

            What dependency? The error doesn’t say. But I resolved a similar issue when I tried to build the project for iOS, at which point I got an error that told me exactly what was missing.

  • I got this working on macOS after 10+ hours of trying. Here’s my documentation of what I did: https://github.com/accord-net/framework/issues/799#issuecomment-324157507.

    Curiously, unless some of the items I mention in the bottom bullets were necessary, it turned out to be the steps from this post, but without needing System.Drawing.dll. I originally got thrown off because I couldn’t find System.Drawing.dll on my computer, but that link above also points to where I eventually found it (though I didn’t keep it in Assets).

  • there is some updated tutorial here,
    Installing the Accord.NET Framework in Unity
    https://github.com/accord-net/framework/wiki/Unity

  • U really expect me to code to run Accord.Net ? Madman’s developers developing Libraries to be used only by developers. No install packages. Why don’t u say so loud and clear. Why i have to install oversize’d software packages in order to sun few lines of code mainly CLI driven with commands that needs to be learn first ? Why i have to code it in order to be able to run it ? If this approach would be used from Microsoft then no user will use it. That’s why such packages and Libraries are widely learned and used !? U cant even start to learn about ANN’s what so ever if u cant run it ? Right ?

Leave a comment to mgear

Connect

Twitter View LinkedIn profile Youtube Youtube Join Discord Twitch

UnityLauncherPro

Get UnityLauncherPRO and work faster with Unity Projects!
*free unity hub alternative

@unitycoder_com

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.