ok

Zenbo SDK - Getting Started

System Requirement

Zenbo is based on Android M. You can use Android Studio or other IDE for android developing. Please refer to Android Studio for more information.

SDK package

  1. Java Jar file (ZenboSDK.jar)
  2. Simple library with extended Android Activity
  3. RobotDevExample source code
  4. Javedoc for ZenboSDK

SDK equipment

  1. Android M, API level 23
  2. Dependency Google GSON library

 

About Zenbo SDK

DS: Dialogue System

CSR: Continuous Speech Recognition

SLU: Spoken Language Understanding

APP bring up

Prepared by developer:

  1. Developer adds cross intent to the DS editor, such as “take a photo”, then edit your package name and launch Activity name.
  2. Use the DS editor to define app actions so that you can control the app by using voice commands.
  3. Develop your app by using the Zenbo SDKs and include subclass motion, vision, robot, and utility.

Executed by user:

  1. Use the voice command “Hey Zenbo” to open the CSR, and then tell the cross intent (e.g. take a photo).
  2. The DS system sends the voice command to the Cloud and returns the SLU result.
  3. The robot framework opens the app by parsing the SLU result.
  4. App receives in-app SLU result that was defined in the DS editor by using Zenbo SDKs and performs the corresponding action.
  5. APP can access the robot’s functions by the Zenbo SDKs.

Zenbo SDK Architecture

Subclass

The Zenbo SDK includes the following subclass:

  1. Robot: Dialog system, Expression.
  2. Vision: Face detect, Body tracing, Arm gesture, Measure Height.
  3. Motion: Move Body/Head, Remote control.
  4. Utility: Follow user, Go there by gesture, Play emotional action.
  5. WheelLights: Blinking, Breathing, Charging, Marquee.
  6. Contacts: User profile, Room info
  7. Slam: Get localization

Callback

Every function returns a command serial number. Developer can use the serial number to verify the result.

  • General callback
    • onStateChange: return command status, includes ACTIVE, PENDING, SUCCESS, FAIL. If the callback returns ”FAIL”, the parameter contains an error code.
    • onResult: return parameter when command process.
  • DS callback: Listen class
    • onResult: SLU result.
    • onRetry: automatically ask again when the DS can’t recognize the result.
    • onVoiceDetect: detect voice event, e.g. detect HeyZenbo, Start/stop CSR
  • Vision callback
    • onDetectFaceResult: return face detect result, includes face location, face box location, etc
    • onDetectPersonResult: return person location.
    • onGesturePoint: return arm gesture point position.

SLU example

{
    "system_info": {
        "version_info": "ver1.0"
    },
    "event_slu_query": {
        "user_utterance": [
            {
                "CsrType": "Google",
                "result": [
                    "show me a",
                    "show me a call",
                    "show me",
                    "show me all",
                    "show me a song"
                ]
            },
            {
                "CsrType": "vocon",
                "result": "show me the photo"
            }
        ],
        "correctedSentence": "show me the photo",
        "error_code": "success",
        "app_semantic": {
            "IntentionId": "gallery.photo.request",
            "Domain": "49",
            "domain": "com.asus.robotgallery",
            "CrossIntent": true,
            "output_context": [
                "gallery_device_previous_state",
                "gallery_device_choose",
                "gallery_device_choose_number",
                "gallery_cancel",
                "gallery_quit",
                "gallery_show_tutorial",
                "gallery_repeat_tts"
            ],
            "Phrase": []
        },
        "speaker_id": "",
        "doa": 0
    }
}

Please refer to the DS document for more information.

 

Important notes on Zenbo APP developing

Manifest file

Zenbo SDK requires the following tags:

<application
        android:allowBackup="true"
        android:icon="@mipmap/ic_launcher"
        android:label="@string/app_name"
        android:supportsRtl="true"
        android:theme="@style/AppTheme">
    <meta-data android:name="zenbo_ds_domainuuid" android:value="82F199B9E7774C688114A72457E3C223"/>
    <meta-data android:name="zenbo_ds_version_82F199B9E7774C688114A72457E3C223" android:value="0.0.1" />
    <activity
                android:name=".MainActivity"
                ...
            
        <intent-filter>
            <action android:name="android.intent.action.MAIN" />
            <category android:name="android.intent.category.LAUNCHER" />
            <category android:name="com.asus.intent.category.ZENBO" />
            <category android:name="com.asus.intent.category.ZENBO_LAUNCHER" />
            <data android:name="com.asus.intent.data.MIN_ROBOT_API_LEVEL.1" />
        </intent-filter>
    </activity>
</application>

Note:

<meta-data android:name="zenbo_ds_domainuuid" android:value="82F199B9E7774C688114A72457E3C223"/> and
<meta-data android:name="zenbo_ds_version_82F199B9E7774C688114A72457E3C223" android:value="0.0.1" />:
It is DDE definition. You must write the information in Manifest, or voice command can’t open app. “2F199B9E7774C688114A72457E3C223” is domain UUID. “0.0.1” is the version of your DDE table. Please reference DDE guild for detail.

<category android:name="android.intent.category.LAUNCHER" />:
We recommend removing this line. When this line is removed, the user can’t open apps by touch and is only allowed to open apps by using voice commands.

<category android:name="com.asus.intent.category.ZENBO" />:
This line declares that this activity is compatible with Zenbo SDK. This category is used by the Zenbo app to list the compatible apps installed on the robot.

<category android:name="com.asus.intent.category.ZENBO_LAUNCHER" />:
This line allows your app to show up in the Zenbo Launcher.

<data android:name="com.asus.intent.data.MIN_ROBOT_API_LEVEL.1" />:
An integer designating the minimum robot API Level required for the application to run. Zenbo store will prevent the user from installing the application if the Robot API Level is lower than the value specified in this attribute. It is similar to Android API Level.

Get cross-intent SLU form intent at APP startup

Intent intent = getIntent();
JSONObject slu_json;
try {
    slu_json = new JSONObject(intent.getStringExtra("json"));
} catch (NullPointerException | JSONException ex) {
    ex.printStackTrace();
}

OnCreate - declare RobotAPI

RobotAPI mRobotAPI;

@Override
protected void ononCreate (Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    mRobotAPI  = new RobotAPI(thisContext, callbackHandle);
}

onPause/onResume:

@Override
protected void onResume() {
    super.onResume();
    mRobotAPI.robot.registerListenCallback(dsCallback);
}
@Override
protected void onPause() {
    super.onPause();
    mRobotAPI.robot.unregisterListenCallback();
}

Note:

When the app is in the “onPause” status, callback will still work. The app needs to call unregisterListenCallback to block the connection. If the app has previously blocked the connection and the app status is changed back to “onResume”, the app needs to call registerListenCallback to reconnect the link.

Navigation and status bar

We recommend hiding the navigation and status bar for the UI.

Extend RobotActivity

RobotActivity is the starting point for coding an app using the Zenbo SDK. RobotActivity is the base activity that provides easy integration with Zenbo, and is commonly required when creating an activity for Zenbo. Please refer to the example app.

 

Access Robot Sensor

Introduction

The robot sensor access is integrated into android SensorManager.
Just use android SensorManager to get the sensor values.

Sensor Type

Capacity touch sensor:

There is a touch sensor on Zenbo’s head. This sensor can sense touch and can identify touch times as a “short touch” or a “long touch”.

Drop laser sensor:

Zenbo has five drop sensors in its base. These sensors can help sense the distance of the ground around Zenbo’s base.

Sonar:

Zenbo uses sonar to measure the distance of its surroundings. Each wheel has one sonar sensor. Zenbo also has three sensors located at the front of its body, as well as one in the rear.

Odometry:

Odometry provides current location and direction based on wheel count. When Zenbo powers on the odometry resets the location to (0, 0) and its direction to 0 rad.
This can help calculate the relative position between different time points..

Neck encoder:

This encoder is used to monitor Zenbo’s neck angle

Wheel encoder:

This encoder is used to monitor wheel speed

Body accelerometer raw data:

This sensor reports the accelerometer data in the body.

Body gyroscope raw data:

This sensor reports the gyroscope data in the body.

Motor:

This sensor reports the data of the motors.

Dock IR:

Reports the data of the IR sensors in the back of the body.

Neck trajectory:

Report the current trajectory.

Wheel trajectory:

Report the current wheel trajectory.

Sensor List

Sensor

SensorEvent Data

Description

Units

TYPE_DROP_LASER

SensorEvent.values[0]

Drop laser info #5

meters

SensorEvent.values[1]

Drop laser info #4

meters

SensorEvent.values[2]

Drop laser info #3

meters

SensorEvent.values[3]

Drop laser info #2

meters

SensorEvent.values[4]

Drop laser info #1

meters

SensorEvent.values[5]

Drop laser info #5

mcps

SensorEvent.values[6]

Drop laser info #4

mcps

SensorEvent.values[7]

Drop laser info #3

mcps

SensorEvent.values[8]

Drop laser info #2

mcps

SensorEvent.values[9]

Drop laser info #1

mcps

TYPE_CAPACITY_TOUCH

SensorEvent.values[0]

1: 0.35s > T >= 0.005s
2: 1s > T >= 0.35s
3: 3s > T >= 1s
4: T > 3s

 

TYPE_SONAR

SensorEvent.values[0]

Right sonar

 

SensorEvent.values[1]

Left sonar

 

SensorEvent.values[2]

Back sonar

 

SensorEvent.values[3]

Right sonar on the front side

 

SensorEvent.values[4]

Left sonar on the front side

 

SensorEvent.values[5]

Central sonar on the front side

 

TYPE_ODOMETRY

SensorEvent.values[0]

The current position of the base in x axis

meters

SensorEvent.values[1]

The current position of the base in y axis

meters

SensorEvent.values[2]

The heading direction of the base

radian

TYPE_NECK_ENCODER

SensorEvent.values[0]

Robot neck yaw

radian

SensorEvent.values[1]

Robot neck pitch

radian

SensorEvent.values[2]

Bit 0 : Busy Yaw
Bit 1 : Busy Pitch
Bit 2 : Error Yaw (over stop position)
Bit 3 : Error Pitch (over stop position)
Bit 4 : Break Yaw
Bit 5 : Break Pitch
Bit 6 : Driver Yaw
Bit 7 : DriverPitch

 

TYPE_WHEEL_ENCODER

SensorEvent.values[0]

Left wheel

m/s

SensorEvent.values[1]

Right wheel

m/s

TYPE_ROBOT_BODY_ACCELEROMETER_RAW

SensorEvent.values[0]

Robot body acceleration force along the x axis(including gravity)

m/s2

SensorEvent.values[1]

Robot body acceleration force along the y axis(including gravity)

m/s2

SensorEvent.values[2]

Robot body acceleration force along the z axis(including gravity)

m/s2

TYPE_ROBOT_BODY_GYROSCOPE_RAW

SensorEvent.values[0]

Robot body rate of rotation around the x axis

rad/s

SensorEvent.values[1]

Robot body rate of rotation around the y axis

rad/s

SensorEvent.values[2]

Robot body rate of rotation around the z axis

rad/s

TYPE_ROBOT_MOTOR

SensorEvent.values[0]

Neck yaw current

A

SensorEvent.values[1]

Neck pitch current

A

SensorEvent.values[2]

Left wheel current

A

SensorEvent.values[3]

Right wheel current

A

SensorEvent.values[4]

Neck yaw PWM

counts

SensorEvent.values[5]

Neck pitch PWM

counts

SensorEvent.values[6]

Left wheel PWM

counts

SensorEvent.values[7]

Right wheel PWM

counts

TYPE_ROBOT_DOCK_IR

SensorEvent.values[0]

Right dock IR

 

SensorEvent.values[1]

Center dock IR

 

SensorEvent.values[2]

Left dock IR

 

TYPE_ROBOT_NECK_TRAJECTORY

SensorEvent.values[0]

Robot neck yaw

radian

SensorEvent.values[1]

Robot neck

radian

TYPE_ROBOT_WHEEL_TRAJECTORY

SensorEvent.values[0]

Left wheel

m/s

SensorEvent.values[1]

Right wheel

m/s

How To Use The Sensors

Sensor type parameters are declared in RobotAPI.Utility.SensorType.

Please refer to the chapter named “Understanding RobotSensorSample APP” and the “RobotSensorSample” source code for details.

 

Understanding RobotDevExample APP

Each app includes 3 modules:

  1. ZenboSDK: module of Zenbo SDK.
  2. RobotActivityLibrary: AAR module with extended activity and imported Zenbo SDK.
  3. RobotDevSample-RobotDevSample: sample code for Zenbo SDK.

Motion example: MotionMoveBodyHead.java

This is an example to use the Zenbo SDK. You can check the option motion->moveBody & moveHead.

You can enter the x, y, and theta values to let Zenbo move its body.

mBtn_MoveBody.setOnClickListener(new Button.OnClickListener() {
    @Override
    public void onClick(View v) {
        float x = Float.valueOf(mEditTextMoveBodyX.getText().toString());
        float y = Float.valueOf(mEditTextMoveBodyY.getText().toString());
        float theta = Float.valueOf(mEditTextMoveBodyTheta.getText().toString());
        robotAPI.motion.moveBody(x, y, theta);
    }
});

You can also set the speed level.

mBtn_moveBodyLevel.setOnClickListener(new Button.OnClickListener() {
    @Override
    public void onClick(View v) {

        float x = Float.valueOf(mEditTextMoveBodyLevelX.getText().toString());
        float y = Float.valueOf(mEditTextMoveBodyLevelY.getText().toString());
        float theta = Float.valueOf(mEditTextMoveBodyLevelTheta.getText().toString());
        int selectedLevel = mSpinnerMoveBodyLevel.getSelectedItemPosition() + 1;
        MotionControl.SpeedLevel.Body level = MotionControl.SpeedLevel.Body.getBody(selectedLevel);

        robotAPI.motion.moveBody(x, y, theta, level);

    }
});

moveHead is similar to moveBody. You can enter the yaw, pitch, and speed level.

mBtn_moveHead.setOnClickListener(new Button.OnClickListener() {
    @Override
    public void onClick(View v) {
        String tmp = mEditText_head_pitch.getText().toString();
        float pitch = TextUtils.isEmpty(tmp) ? 0 : (float) Math.toRadians(Float.valueOf(tmp));

        tmp = mEditText_head_yaw.getText().toString();
        float yaw = TextUtils.isEmpty(tmp) ? 0 : (float) Math.toRadians(Float.valueOf(tmp));

        int selectedLevel = mSpinnerMoveHeadLevel.getSelectedItemPosition() + 1;
        MotionControl.SpeedLevel.Head level = MotionControl.SpeedLevel.Head.getHead(selectedLevel);

        robotAPI.motion.moveHead(yaw, pitch, level);
    }
});

And you can get the result by callback.

public static RobotCallback robotCallback = new RobotCallback() {
    @Override
    public void onStateChange(int cmd,int serial,RobotErrorCode err_code,RobotCmdState state) {
        super.onStateChange(cmd, serial, err_code, state);
    }
}

Finally, you can stop the motion by the stopping function.

Button stop = (Button) findViewById(R.id.stopButton);
stop.setOnClickListener(new Button.OnClickListener() {
    @Override
    public void onClick(View v) {
        robotAPI.motion.stopMoving();
    }
});

Speak example: RobotSpeak.java

Zenbo can speak some words by your inputs. Same with motions, you can get successful callback from onStateChange.

mEdit = (EditText)findViewById(R.id.speak_edittext);

btn_start_speak.setOnClickListener(new Button.OnClickListener(){
    @Override
    public void onClick(View v) {
        robotAPI.robot.speak(mEdit.getText().toString());
    }
});

And you can stop Zenbo from talking when it is speaking.

btn_stop_speak.setOnClickListener(new Button.OnClickListener(){
    @Override
    public void onClick(View v) {
        robotAPI.robot.stopSpeak();
    }
});

Expression example: RobotSetExpression.java

Zenbo can change its facial expressions and has 24 different faces for you to choose from.

if(SpinnerText.equals("INTEREST")){
    robotAPI.robot.setExpression(RobotFace.INTEREST);
}
else if(SpinnerText.equals("DOUBT")){
    robotAPI.robot.setExpression(RobotFace.DOUBT);
}
else if(SpinnerText.equals("PROUD")){
    robotAPI.robot.setExpression(RobotFace.PROUD);
}
else if(SpinnerText.equals("DEFAULT")){
    robotAPI.robot.setExpression(RobotFace.DEFAULT);
}
else if(SpinnerText.equals("HAPPY")){
    robotAPI.robot.setExpression(RobotFace.HAPPY);
}

PlayAction example: UtilityPlayAction.java

Zenbo has several canned action to express emotions.

mEditText.setText("22");      //#22 action

btn_start.setOnClickListener(new View.OnClickListener() {
    @Override
    public void onClick(View arg0) {

        if(mEditText.getText().length() > 0) {
            int iCuttentNumberPickerValue = Integer.parseInt(mEditText.getText().toString());
            robotAPI.utility.playAction(iCuttentNumberPickerValue); 
        }

    }
});

Some actions are on looping. You can stop the loop by a cancel command.

btn_stop.setOnClickListener(new View.OnClickListener() {
    @Override
    public void onClick(View arg0) {
        robotAPI.cancelCommand(RobotCommand.MOTION_PLAY_ACTION.getValue());
    }
});

Emotional example: UtilityPlayEmotionalAction.java

Zenbo also has a function that combines face and canned action. You can also set the face to be randomized. Example 1 below is a canned action with 5 possible faces picked at random.

if(SpinnerText.equals("Example 1")){
    List<RobotUtil.faceItem> faceItemList = new ArrayList<>();
    faceItemList.add(new RobotUtil.faceItem(RobotFace.DEFAULT, 10));
    faceItemList.add(new RobotUtil.faceItem(RobotFace.HAPPY, 10));
    faceItemList.add(new RobotUtil.faceItem(RobotFace.EXPECT, 10));
    faceItemList.add(new RobotUtil.faceItem(RobotFace.SHOCK, 10));
    faceItemList.add(new RobotUtil.faceItem(RobotFace.LAZY, 10));

    iCurrentCommandSerial = robotAPI.utility.playEmotionalAction(faceItemList, 5);
}

Detect person example: VisionRequestDetectPerson.java

Zenbo can detect a person and return callback.

private void detectPersonClicked() {
    robotAPI.vision.requestDetectPerson( 1 );
}

And the result will report the location by callback: onDetectPersonResult.

@Override
public void onDetectPersonResult(List<DetectPersonResult> resultList) {
    super.onDetectPersonResult(resultList);
    if (resultList.size() == 0)
        Log.d("RobotDevSample", "onDetectPersonResult: empty");
    else
        Log.d("RobotDevSample", "onDetectPersonResult: " + resultList.get(0).getBodyLoc().toString());
}

WheelLights Example: WheelLightsActivity.java

Zenbo has two side wheel LEDs. You can control the LED brightness, color, and pattern.

case 2:
    //.startBlinking
    robotAPI.wheelLights.turnOff(WheelLights.Lights.SYNC_BOTH, 0xff);
    robotAPI.wheelLights.setColor(WheelLights.Lights.SYNC_BOTH, 0xff, 0x007F7F);
    robotAPI.wheelLights.setBrightness(WheelLights.Lights.SYNC_BOTH, 0xff, 10);
    robotAPI.wheelLights.startBlinking(WheelLights.Lights.SYNC_BOTH, 0xff, 30, 10, 5);
    break;

case 3:
    //.startBreathing
    robotAPI.wheelLights.turnOff(WheelLights.Lights.SYNC_BOTH, 0xff);
    robotAPI.wheelLights.setColor(WheelLights.Lights.SYNC_BOTH, 0xff, 0x00D031);
    robotAPI.wheelLights.setBrightness(WheelLights.Lights.SYNC_BOTH, 0xff, 10);
    robotAPI.wheelLights.startBreathing(WheelLights.Lights.SYNC_BOTH, 0xff, 20, 10, 0);
    break;

case 4:
    //.startCharging
    robotAPI.wheelLights.turnOff(WheelLights.Lights.SYNC_BOTH, 0xff);
    robotAPI.wheelLights.setColor(WheelLights.Lights.SYNC_BOTH, 0xff, 0xFF9000);
    robotAPI.wheelLights.setBrightness(WheelLights.Lights.SYNC_BOTH, 0xff, 10);
    robotAPI.wheelLights.startCharging(WheelLights.Lights.SYNC_BOTH, 0, 1, WheelLights.Direction.DIRECTION_FORWARD, 20);
    break;
case 5:
    //.startMarquee
    robotAPI.wheelLights.turnOff(WheelLights.Lights.SYNC_BOTH, 0xff);
    robotAPI.wheelLights.setBrightness(WheelLights.Lights.SYNC_BOTH, 0xff, 20);
    robotAPI.wheelLights.startMarquee(WheelLights.Lights.SYNC_BOTH, WheelLights.Direction.DIRECTION_FORWARD, 40, 20, 14);
    break;

 

Understanding RobotSensorSample APP

Declare Sensor Example: MainActivity.java

Declare the sensor name to through Utility.SensorType. This step also can be ignored, just using Utility.SensorType to register sensor.

public static final int TYPE_CAPACITY_TOUCH = Utility.SensorType.CAPACITY_TOUCH;
public static final int TYPE_DROP_LASER = Utility.SensorType.DROP_LASER;
public static final int TYPE_SONAR = Utility.SensorType.SONAR;
public static final int TYPE_ODOMETRY = Utility.SensorType.ODOMETRY;
public static final int TYPE_NECK_ENCODER = Utility.SensorType.NECK_ENCODER;
public static final int TYPE_WHEEL_ENCODER = Utility.SensorType.WHEEL_ENCODER;
public static final int TYPE_ROBOT_BODY_ACCELEROMETER_RAW = Utility.SensorType.ROBOT_BODY_ACCELEROMETER_RAW;
public static final int TYPE_ROBOT_BODY_GYROSCOPE_RAW = Utility.SensorType.ROBOT_BODY_GYROSCOPE_RAW;
public static final int TYPE_ROBOT_MOTOR = Utility.SensorType.ROBOT_MOTOR;
public static final int TYPE_ROBOT_DOCK_IR = Utility.SensorType.ROBOT_DOCK_IR;
public static final int TYPE_ROBOT_NECK_TRAJECTORY = Utility.SensorType.ROBOT_NECK_TRAJECTORY;
public static final int TYPE_ROBOT_WHEEL_TRAJECTORY = Utility.SensorType.ROBOT_WHEEL_TRAJECTORY;

Create Sensor Listener Example: MainActivity.java

Create the sensor event listener to prepare to receive sensor data. This is using android SensorEventListener, please refer the android official guide to get more details.

SensorEventListener listenerCapacityTouch = new SensorEventListener() {
    @Override
    public void onSensorChanged(SensorEvent event) {

        mTextView_capacity_touch_value0.setText(String.valueOf(event.values[0]));
        mTextView_capacity_touch_value1.setText(String.valueOf(event.values[1]));

    }

 

Q&A

  1. Why does my APP close and return back to Zenbo’s face automatically?

    Zenbo has two modes: Android mode and Zenbo mode.
    1. Android mode: Press the home button when Zenbo’s face has appeared.
    2. Zenbo mode: Zenbo’ face appears on the screen. You can open Apps using voice commands or Zenbo Apps.

    You must open the Zenbo app in Zenbo mode; otherwise it will return to Zenbo’s face after 1 minute. Adding FLAG_KEEP_SCREEN_ON can bypass Zenbo modes timeout, but it will still return to Zenbo's face when the sound of “Hey Zenbo” is received.
     
  2. Why can I not find my newest app on the Zenbo store?

    Please check the MIN_ROBOT_API_LEVEL. Zenbo store will display the apps with a higher ROBOT_API_LEVEL than system image has. You can assign a lower MIN_ROBOT_API_LEVEL in the manifest if you don’t use the newer API. Most of APIs are backwards compatible.
     
  3. Can I use the head press function?

    Yes this can be done in two steps.
    1. Close system behavior by using RobotAPI.robot.setPressOnHeadAction.
    2. Reference AndroidSensorManager with Utility.SensorType.CAPACITY_TOUCH type to listen for the “pressing on head” event callback.
       
  4. Can I get the 3D camera steam or depth view raw data?

    No, 3D camera is not available to developer currently.
     
  5. What does VisionConfig mean?

    We have added the description for each parameter. Please refer to the latest ZenboSDK document for more information.
     
  6. “Hey Zenbo” and looking at the user doesn’t work in noisy environments. What can I do?

    You can use Bluetooth microphone instead. Unfortunately, Zenbo cannot get the direction of sound from Bluetooth microphone.
     
  7. Why am I getting SERVICE_FAIL when I use the APIs?

    Please use the new ZenboSDK instance in onCreate and use the APIs after receving onResume or RobotCallback.initComplete(). Do not use any Robot API in the function which creates ZenboSDK instance.
     
Go To Top