This is Our Blog

August 10, 2015
|

Arduino control by Kinect on local or network machine.


In this tutorial we are going learn and create a demo project of “Arduino control by Kinect on local or network machine“. We will go through step by step.

STEP-1. Networking Concept

Create a new project–> Create a new scene named “scnDemo”, then implement networking in project.

Please read our previous blog on Unity Networking Concept.

http://wordpress-devinit-resources-india.azurewebsites.net/virtual-meeting-networking-concept/

STEP- 2. Implement Kinect

Please read our blog on “How to setup Kinect with Unity”.

http://wordpress-devinit-resources-india.azurewebsites.net/how-to-setup-kinect-with-unity/

After successfully importing Kinect assets.

  1. Go to “Assets / AvatarsDemo” folder.
  2. Open “KinectAvatarsDemo” scene. There are two humanoid models by default. You can change default humanoid models with your own humanoid model.
  3. Copy U_CharacterFront and Main Camera objects.
  4. Go to “scnDemo” –> Hierarchy –> Paste
  5. Rename Main Camera to Kinect Camera and U_CharacterFront to Player
  6. Go to Prefabs folder–> delete old Player prefab and drag new Player inside folder from scene Hierarchy.
  7. Now go to Network Manager Script (Attached with scnDemo) –> Player prefab –> drag and attach Player gameobject from Prefabs folder.

STEP- 3. Implement Uniduino Asset

Please read our blog “How to setup Arduino board with unity”.

http://wordpress-devinit-resources-india.azurewebsites.net/how-to-setup-arduino-board-in-unity/

  1. Go to “Assets/Uniduino/Prefab” folder
  2. Copy Uniduino and paste on scnDemo scene (Hierarchy).
  3. Enter your Port Name of Arduino Script.


  4. Now open Network Manager Script in your editor. Replace all codes with below codes.

using UnityEngine;
using System.Collections;

public class NetworkManager : MonoBehaviour 
{ 
    private const string typeName = "ArduinoKinectDemo";
    private const string gameName = "Person";
    private bool isRefreshingHostList = false;
    private HostData[] hostList;
    public GameObject playerPrefab;
    public GUISkin skin;
    bool startBtn = true;
    static bool timerStart = false;
    static float timespan = 0;
    static bool timerStart_kinect = false;
    static float timespan_kinect = 0;
    static string btnName = "Start Meeting";
    FontStyle ft;
    public GUISkin gui_skin;
    GUIStyle lbl;
    public Camera Kinect_cam;
    public Camera cam;
    public void OnGUI()
    {
        GUI.skin = gui_skin;
        GUILayout.BeginArea(new Rect(0f, 0f, Screen.width, Screen.height));

        lbl = gui_skin.label;
        lbl.fontSize = 22;
        lbl.alignment = TextAnchor.MiddleCenter;

        GUI.Label(new Rect(Screen.width / 5.5f, (Screen.height / 5.5f), Screen.width / 1.75f, 50), "<color=#008080ff><b>DEViNIT</b></color> VIRTUAL MEETING PLACE", lbl);// Start Server             
        if (!Network.isClient && !Network.isServer)
        {

            if (startBtn)
            {
                if (GUI.Button(new Rect(Screen.width / 2.5f, (Screen.height / 3f) + 100, 250, 100), btnName)) // Start Server
                {
                    timerStart = true;
                    btnName = "Please Wait";
                    RefreshHostList();

                }

                if ((System.Math.Round(timespan) > 0.9f) && (timerStart == true))
                {
                    bool st = false;
                    if (hostList != null)
                    {
                        print("yes");
                        for (int i = 0; i < hostList.Length; i++)
                        {
                            st = true;
                            startBtn = true;
                        }
                    }
                    else
                    {
                        print("no");
                        StartServer();
                    }
                    if (st)
                    {
                        print("no");
                        st = false;
                        startBtn = false;
                        StartServer();
                    }
                }
            }

            else
            {
                GUI.Label(new Rect(Screen.width / 2.5f, (Screen.height / 3f) + 100, 250, 100), "Connect ->");// Start Server             
            }

            if (GUI.Button(new Rect(Screen.width / 2.5f, (Screen.height / 3f) + 250, 250, 100), "Refresh Persons")) // Refresh Persons
            {
                RefreshHostList();
            }

            if (hostList != null)
            {
                for (int i = 0; i < hostList.Length; i++)
                {
                    startBtn = false;
                    timerStart = false;
                    if (GUI.Button(new Rect(Screen.width / 1.8f, (Screen.height / 3f) + 100 + (110 * i), 300, 100), hostList[i].gameName))
                    {
                        JoinServer(hostList[i]);
                        timerStart_kinect = true;
                    }
                }
            }
        }
        GUILayout.EndArea();

        //_____________________________Start kinect for person click___________________
        if ((System.Math.Round(timespan_kinect) > 3.9f) && (timerStart_kinect == true))
        {
            Kinect_cam.active = true;
            cam.active = false;
            timerStart_kinect = false;
        }

    }

    void OnMasterServerEvent(MasterServerEvent msEvent)
    {
        if (msEvent == MasterServerEvent.RegistrationSucceeded)
            Debug.Log("Server registered");

        if (msEvent == MasterServerEvent.HostListReceived)
        {
            if (btnName == "Please Wait")
            {
                Kinect_cam.active = true;
                cam.active = false;
            }
        }

        if (msEvent == MasterServerEvent.RegistrationFailedGameType)
            Debug.Log("Error");



    }

    private void StartServer()
    {

        Network.InitializeServer(5, 25000, !Network.HavePublicAddress());
        MasterServer.RegisterHost(typeName, gameName);

        startBtn = false;

    }

    void OnServerInitialized()
    {
        SpawnPlayer();
        print("server start");
    }


    void Update()
    {
        if (timerStart)
        {
            timespan = timespan + Time.deltaTime;
        }
        if (timespan > 1)
        {
            timerStart = false;
        }

        if (timerStart_kinect)
        {
            timespan_kinect = timespan_kinect + Time.deltaTime;
        }
        if (timespan_kinect > 4)
        {
            timerStart_kinect = false;
        }


        if (isRefreshingHostList && MasterServer.PollHostList().Length > 0)
        {
            isRefreshingHostList = false;
            hostList = MasterServer.PollHostList();
        }
    }

    private void RefreshHostList()
    {
        if (!isRefreshingHostList)
        {
            isRefreshingHostList = true;
            MasterServer.RequestHostList(typeName);
        }
    }

    private void JoinServer(HostData hostData)
    {
        Network.Connect(hostData);
    }

    void OnConnectedToServer()
    {
        SpawnPlayer();
    }

    static float margin = 1f;
    private void SpawnPlayer()
    {
        margin = margin + 2f;
        Network.Instantiate(playerPrefab, new Vector3(0f, 0f), new Quaternion(0,180f,0,0), 0);
    }
}

  1. Now go to scene and set values of fields.


STEP- 4. Configure Circuit Sketch

Requirement:-

  • Arduino Uno board – 1
  • Breadboard – 1
  • LED – 1
  • 220 Ohm Resistors – 2
  • NPN Transistor – 1
  • Rectifier Diode – 1
  • DC Motor – 1
  • Jumper wires


STEP- 5. Creating Users Signal Panel

We will create two Panels, one for local user signal and another for Remote user signals.

Signal Panel for Local User

  1. Use –> 1 Quad, 4 cubes and 1 Spheres.
  2. Go to GameObject–> Create Empty named “Arduino1–> Put all above game objects inside Arduino1, like below image yellow square area.
  3. Arrange their position and size according to below image or your choice.


  1. Add a new tag called “Ard1”. Change tags of all child game objects or Arduino1 to “Ard1”.

Signal Panel for Local User

  1. Use –> 1 Quad, 3 cubes and 1 Spheres.
  2. Go to GameObject–> Create Empty named “Arduino2–> Put all above game objects inside Arduino2, like below image yellow square area.
  3. Arrange their position and size according to below image or your choice.


  1. Add a new tag called “Ard2”. Change tags of all child game objects or Arduino2 to “Ard2”.

STEP- 6. Update Player (Synchronization through network)

  1. Create a C# Script named “PlayerArdino”.
  2. Open script in your editor and replace all by below codes
using UnityEngine;
using System.Collections;
using Uniduino;

public class PlayerArdino : MonoBehaviour
{
    private float lastSynchronizationTime = 0f;
    private float syncDelay = 0f;
    private float syncTime = 0f;
    private Vector3 syncStartPosition = Vector3.zero;
    private Vector3 syncEndPosition = Vector3.zero;
    private bool isBlink;
    private bool isMotorRun;

    void OnSerializeNetworkView(BitStream stream, NetworkMessageInfo info)
    {
        Vector3 syncPosition = Vector3.zero;
        Vector3 syncVelocity = Vector3.zero;
        bool _isBlink = false;
        bool _isMotorRun = false;

        if (stream.isWriting)
        {
            syncPosition = transform.position;
            stream.Serialize(ref syncPosition);

            _isBlink = isBlink;
            stream.Serialize(ref _isBlink);

            _isMotorRun = isMotorRun;
            stream.Serialize(ref _isMotorRun);

        }
        else
        {
            stream.Serialize(ref syncPosition);
            syncStartPosition = transform.position;

            stream.Serialize(ref _isBlink);
            isBlink = _isBlink;

            stream.Serialize(ref _isMotorRun);
            isMotorRun = _isMotorRun;
        }
    }

    void Awake()
    {
        lastSynchronizationTime = Time.time;
    }
    public Arduino arduino;
    int pinLED = 13;
    int pinMotor = 9;
    public GameObject objLED;
    public GameObject objMotor;
    public GameObject objLED2;
    public GameObject objMotor2;

    GameObject[] objArd1;
    GameObject[] objArd2;
    int speed = 0;
    public static bool isMove = false;
    public static bool isArduinoMine = true;
    void Start()
    {
        objArd1 = (GameObject[])GameObject.FindGameObjectsWithTag("Ard1");
        objArd2 = (GameObject[])GameObject.FindGameObjectsWithTag("Ard2");
        foreach (GameObject gObject in objArd1)
        {
            if (gObject.name == "BlinkSphere")
            {
                objLED = gObject;
            }
            if (gObject.name == "WingCube")
            {
                objMotor = gObject;
            }
        }
        foreach (GameObject gObject in objArd2)
        {
            if (gObject.name == "BlinkSphere2")
            {
                objLED2 = gObject;
            }
            if (gObject.name == "WingCube2")
            {
                objMotor2 = gObject;
            }
        }
        arduino = Arduino.global;
        arduino.Setup(ConfigurePins);
        StartCoroutine(BlinkLED());
        StartCoroutine(RunMotor());
    }

    void ConfigurePins()
    {
        arduino.pinMode(pinLED, PinMode.OUTPUT);
        arduino.pinMode(pinMotor, PinMode.OUTPUT);
    }

    IEnumerator BlinkLED()
    {
        while (true)
        {
            if (networkView.isMine && isArduinoMine)
            {
                if (isBlink)
                {
                    arduino.digitalWrite(pinLED, Arduino.HIGH);
                    objLED.renderer.material.color = Color.red;
                    yield return new WaitForSeconds(0.3f);

                    arduino.digitalWrite(pinLED, Arduino.LOW);
                    objLED.renderer.material.color = Color.white;
                    yield return new WaitForSeconds(0.3f);
                }
                else
                {
                    yield return new WaitForSeconds(0.1f);
                }
            }
            else if (!networkView.isMine && !isArduinoMine)
            {
                if (isBlink)
                {
                    arduino.digitalWrite(pinLED, Arduino.HIGH);
                    objLED2.renderer.material.color = Color.red;
                    yield return new WaitForSeconds(0.3f);

                    arduino.digitalWrite(pinLED, Arduino.LOW);
                    objLED2.renderer.material.color = Color.white;
                    yield return new WaitForSeconds(0.3f);
                }
                else
                {
                    yield return new WaitForSeconds(0.1f);
                }
            }
            else
            {
                yield return new WaitForSeconds(0.1f);
            }
        }
    }

    bool isMineMotor = true;
    IEnumerator RunMotor()
    {
        while (true)
        {
            if (networkView.isMine && isArduinoMine)
            {
                if (isMotorRun)
                {
                    arduino.digitalWrite(pinMotor, Arduino.HIGH);
                    speed = 50;
                    objMotor.transform.Rotate(Vector3.forward, speed);
                    yield return new WaitForSeconds(0.01f);
                }
                else
                {
                    arduino.digitalWrite(pinMotor, Arduino.LOW);
                    yield return new WaitForSeconds(0.01f);
                }
            }
            else if (!networkView.isMine && !isArduinoMine)
            {
                if (isMotorRun)
                {
                    arduino.digitalWrite(pinMotor, Arduino.HIGH);
                    speed = 50;
                    objMotor2.transform.Rotate(Vector3.forward, speed);
                    yield return new WaitForSeconds(0.01f);
                }
                else
                {
                    arduino.digitalWrite(pinMotor, Arduino.LOW);
                    yield return new WaitForSeconds(0.01f);
                }
            }
            else
            {
                yield return new WaitForSeconds(0.01f);
            }
        }

    }
    void Update()
    {
        if (networkView.isMine)
        {

            InputMovement();
            InputColorChange();
            print(isBlink);
        }
        else
        {
            //  SyncedMovement();            
        }
    }

    private void InputMovement()
    {
        if (isArduinoMine)
        {
            isBlink = System.Convert.ToBoolean(PlayerPrefs.GetInt("isBlink"));
            isMotorRun = System.Convert.ToBoolean(PlayerPrefs.GetInt("isMotorRun")); ;
        }
    }

    private void SyncedMovement()
    {
        syncTime += Time.deltaTime;
        transform.position = syncEndPosition;
    }

    void OnGUI()
    {
        if (isArduinoMine)
        {
            if (GUI.Button(new Rect(Screen.width / 2.5f, (Screen.height / 3f) + 100, 250, 100), "Arduino by Remote"))
            {
                isArduinoMine = false;
                isBlink = false;
                isMotorRun = false;
            }
        }
        else
        {
            if (GUI.Button(new Rect(Screen.width / 2.5f, (Screen.height / 3f) + 100, 250, 100), "Arduino by Mine"))
            {
                isArduinoMine = true;
                isBlink = false;
                isMotorRun = false;
            }
        }
    }

    private void InputColorChange()
    {

    }

    [RPC]
    void ChangeColorTo(byte[] im)
    {

    }

    void OnApplicationQuit()
    {
        arduino.digitalWrite(pinLED, Arduino.LOW);
        arduino.digitalWrite(pinMotor, Arduino.LOW);
    }
}

  1. Now go to Prefab folder –> Attach PlayerArduino script to Player prefab.
  2. Select Player prefab and click on Add Component button –> Add Network View.
  3. Drag drop PlayerArduino script to Observed option of Network View of Player prefab.


STEP- 7. Kinect interaction with cubes (BlinkCube, MotorCube and StopCube)

We will write/customize code for Kinect interaction with three cubes. We will perform following interaction by grip hand-

  • BlinkCube – Start LED blinking,
  • MotorCube – Start Motor,
  • StopCube – Stop Blinking LED and Motor.

Follow below steps:-

  1. Go to scene –> select Kinect Camera –> Add Component button
    –>
    Add GrabDropScript script.
  2. Now replace Update function with below codes.
void Update()
    {
        manager = InteractionManager.Instance;

        if (manager != null && manager.IsInteractionInited())
        {
            Vector3 screenNormalPos = Vector3.zero;
            Vector3 screenPixelPos = Vector3.zero;

            if (draggedObject == null)
            {
                // no object is currently selected or dragged.
                // if there is a hand grip, try to select the underlying object and start dragging it.
                if (manager.IsLeftHandPrimary())
                {
                    // if the left hand is primary, check for left hand grip
                    if (manager.GetLastLeftHandEvent() == InteractionManager.HandEventType.Grip)
                    {
                        isLeftHandDrag = true;
                        screenNormalPos = manager.GetLeftHandScreenPos();
                    }
                }
                else if (manager.IsRightHandPrimary())
                {
                    // if the right hand is primary, check for right hand grip
                    if (manager.GetLastRightHandEvent() == InteractionManager.HandEventType.Grip)
                    {
                        isLeftHandDrag = false;
                        screenNormalPos = manager.GetRightHandScreenPos();
                    }
                }

                // check if there is an underlying object to be selected
                if (screenNormalPos != Vector3.zero)
                {
                    // convert the normalized screen pos to pixel pos
                    screenPixelPos.x = (int)(screenNormalPos.x * Camera.main.pixelWidth);
                    screenPixelPos.y = (int)(screenNormalPos.y * Camera.main.pixelHeight);
                    Ray ray = Camera.main.ScreenPointToRay(screenPixelPos);

                    // check for underlying objects
                    RaycastHit hit;
                    if (Physics.Raycast(ray, out hit))
                    {
                        foreach (GameObject obj in draggableObjects)
                        {
                            if (hit.collider.gameObject == obj)
                            {
                                draggedObject = obj;
                                // set selection material
                                draggedObjectMaterial = draggedObject.renderer.material;
                                draggedObject.renderer.material = selectedObjectMaterial;

                                if (obj.name == "BlinkCube")
                                {
                                    PlayerPrefs.SetInt("isBlink", 1);
                                }
                                else if (obj.name == "MotorCube")
                                {
                                    PlayerPrefs.SetInt("isMotorRun", 1);

                                }
                                else if (obj.name == "StopCube")
                                {
                                    PlayerPrefs.SetInt("isBlink", 0);
                                    PlayerPrefs.SetInt("isMotorRun", 0);

                                }

                                break;
                            }
                        }
                    }
                }

            }
            else
            {
                // continue dragging the object
                screenNormalPos = isLeftHandDrag ? manager.GetLeftHandScreenPos() : manager.GetRightHandScreenPos();
                                
                // check if the object (hand grip) was released
                bool isReleased = isLeftHandDrag ? (manager.GetLastLeftHandEvent() == InteractionManager.HandEventType.Release) :
                    (manager.GetLastRightHandEvent() == InteractionManager.HandEventType.Release);

                if (isReleased)
                {
                    // restore the object's material and stop dragging the object
                    draggedObject.renderer.material = draggedObjectMaterial;
                    draggedObject = null;
                }
            }
        }
    }

  1. Now return to scene –> Kinect Camera –>
    GrabDropScript script –>

    Draggable Objects

  • Size – 3 (then Press Enter)
  • Element0 — BlinkCube
  • Element1 — MotorCube
  • Element2 – StopCube


Now click on Play button.

Click on Start Meeting button.


If you have done all steps correctly, LED will be blinking. When you connected with Remote user successfully, you can get remote user signals by click on Arduino by Remote button.

Thank you for reading my blog. If you have any query, you can write me mail on neeraj.singh@devinit.ch , it would help us a lot.



0 Comments


Would you like to share your thoughts?

Would you like to share your thoughts?

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.