ffmpeg之native原生绘制

发表于 | 分类于 | |

上次已将ffmpeg的动态库编译出来了，并且使用了ffmpeg的转码功能，成功将mp4格式视频转化为yuv视频，这篇文章基于上次测试的demo，使用surfaceview显示解码完成的像素数据

布局设置和权限添加

布局

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent" >

    <com.cj5785.ffmpegnativeplayer.view.MySurfaceView
        android:id="@+id/surface_view"
        android:layout_width="fill_parent"
        android:layout_height="fill_parent"/>
    
    <Button
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="开始"
        android:onClick="mPlay" />

</FrameLayout>

权限

1
2
3
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.MOUNT_UNMOUNT_FILESYSTEMS" />

编写自定义view和控制器

自定义View

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
package com.cj5785.ffmpegnativeplayer.view;

import android.content.Context;
import android.graphics.PixelFormat;
import android.util.AttributeSet;
import android.view.SurfaceHolder;
import android.view.SurfaceView;

public class MySurfaceView extends SurfaceView {

	public MySurfaceView(Context context) {
		super(context);
		init();
	}
	
	public MySurfaceView(Context context, AttributeSet attrs) {
		super(context, attrs);
		init();
	}
	
	public MySurfaceView(Context context, AttributeSet attrs, int defStyle) {
		super(context, attrs, defStyle);
		init();
	}

	//初始化像素格式
	private void init() {
		SurfaceHolder holder = getHolder();
		holder.setFormat(PixelFormat.RGBA_8888);
	}
}

控制器

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
package com.cj5785.ffmpegnativeplayer;

import android.view.Surface;

public class NativePlayer {
	
	public native void render(String input, Surface surface);
	
	static {
		System.loadLibrary("avutil-54");
		System.loadLibrary("swresample-1");
		System.loadLibrary("avcodec-56");
		System.loadLibrary("avformat-56");
		System.loadLibrary("swscale-3");
		System.loadLibrary("postproc-53");
		System.loadLibrary("avfilter-5");
		System.loadLibrary("avdevice-56");
		System.loadLibrary("ffmpeg_native_player");
	}
	
}

实现控制器native方法

  • 使用javah生成头文件，这里可能存在无法找到Surface签名的问题，这时候需要指定classpath路径
    javah -classpath E:\eclipse-adt\sdk\platforms\android-15\android.jar;. com.cj5785.ffmpegnativeplayer.NativePlayer
    格式说明：-classpath后面跟的是android.jar路径，最后接native方法类的全名

  • 新建jni文件夹，将头文件移至jni文件夹，添加本地依赖

  • 复制生成ffmpeg的include目录和so动态库到jni目录

  • 将之前的Android.mk和Application.mk复制到jni文件夹，并做适当修改
    Android.mk主要修改模块名，使其与控制器调用相统一
    Application.mk主要将APP_PLATFORM := android-8修改为APP_PLATFORM := android-9
    注意，此处如果不修改Application.mk将导致android/native_window_jni.h无法找到，同时，由于使用了这个头文件，需要在Android.mk配置-landroid

  • 使用开源库libyuv实现yuv转化为RGBA_8888
    下载开源库libyuv，下载地址libyuv下载地址
    将libyuv下的所有文件放入jni目录(NDK工程规范，必须存在jni目录)
    修改libyuv的Android.mk文件，将最后的include $(BUILD_STATIC_LIBRARY)改为include $(BUILD_SHARED_LIBRARY)，这样就可以生成so动态库了
    还可以将LOCAL_MODULE := libyuv_static改为LOCAL_MODULE := libyuv，方便so管理
    在jni目录下执行ndk-build即可对libyuv进行编译
    编译生成的so动态库位于与jni目录同级的lib下
    将lib添加到工程jni目录下，为了便于管理，将jni的include目录进行重新分配，重新分配目录如下：(已将libyuv的include加入到工程，这里没有列出目录下包含的头文件)

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    │  Android.mk
    │  Application.mk
    │  com_cj5785_ffmpegnativeplayer_NativePlayer.h
    │  ffmpeg_native_player.c
    │  
    └─include
        ├─ffmpeg
        │  │  libavcodec-56.so
        │  │  libavdevice-56.so
        │  │  libavfilter-5.so
        │  │  libavformat-56.so
        │  │  libavutil-54.so
        │  │  libpostproc-53.so
        │  │  libswresample-1.so
        │  │  libswscale-3.so
        │  │  
        │  ├─libavcodec
        │  ├─libavdevice
        │  ├─libavfilter
        │  ├─libavformat
        │  ├─libavutil
        │  ├─libpostproc
        │  ├─libswresample
        │  └─libswscale
        │          
        └─libyuv
            │  libyuv.h
            │  libyuv.so
            │  
            └─libyuv

  • 修改Android.mk，使其能找到so动态库

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
LOCAL_PATH := $(call my-dir)

include $(CLEAR_VARS)
LOCAL_MODULE := avcodec
LOCAL_SRC_FILES := include/ffmpeg/libavcodec-56.so
include $(PREBUILT_SHARED_LIBRARY)

include $(CLEAR_VARS)
LOCAL_MODULE := avdevice
LOCAL_SRC_FILES := include/ffmpeg/libavdevice-56.so
include $(PREBUILT_SHARED_LIBRARY)

include $(CLEAR_VARS)
LOCAL_MODULE := avfilter
LOCAL_SRC_FILES := include/ffmpeg/libavfilter-5.so
include $(PREBUILT_SHARED_LIBRARY)

include $(CLEAR_VARS)
LOCAL_MODULE := avformat
LOCAL_SRC_FILES := include/ffmpeg/libavformat-56.so
include $(PREBUILT_SHARED_LIBRARY)

include $(CLEAR_VARS)
LOCAL_MODULE := avutil
LOCAL_SRC_FILES := include/ffmpeg/libavutil-54.so
include $(PREBUILT_SHARED_LIBRARY)


include $(CLEAR_VARS)
LOCAL_MODULE := postproc
LOCAL_SRC_FILES := include/ffmpeg/libpostproc-53.so
include $(PREBUILT_SHARED_LIBRARY)

include $(CLEAR_VARS)
LOCAL_MODULE := swresample
LOCAL_SRC_FILES := include/ffmpeg/libswresample-1.so
include $(PREBUILT_SHARED_LIBRARY)

include $(CLEAR_VARS)
LOCAL_MODULE := swscale
LOCAL_SRC_FILES := include/ffmpeg/libswscale-3.so
include $(PREBUILT_SHARED_LIBRARY)

include $(CLEAR_VARS)
LOCAL_MODULE := yuv
LOCAL_SRC_FILES := include/libyuv/libyuv.so
include $(PREBUILT_SHARED_LIBRARY)

include $(CLEAR_VARS)
LOCAL_MODULE := ffmpeg_native_player
LOCAL_SRC_FILES := ffmpeg_native_player.c
LOCAL_C_INCLUDES += $(LOCAL_PATH)/include/ffmpeg
LOCAL_C_INCLUDES += $(LOCAL_PATH)/include/libyuv
LOCAL_LDLIBS := -llog -landroid
LOCAL_SHARED_LIBRARIES := avcodec avdevice avfilter avformat avutil postproc swresample swscale yuv
include $(BUILD_SHARED_LIBRARY)
  • 修改Application.mk，更改APP_PLATFORM，使其可以使用android/native_window_jni.h和android/native_window.h头文件
1
2
APP_ABI := armeabi armeabi-v7a
APP_PLATFORM := android-9
  • 实现jni头文件声明的函数
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
#include <android/log.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <android/native_window.h>
#include <android/native_window_jni.h>

#include "com_cj5785_ffmpegnativeplayer_NativePlayer.h"

//封装格式
#include "include/ffmpeg/libavformat/avformat.h"
//解码
#include "include/ffmpeg/libavcodec/avcodec.h"
//像素处理
#include "include/ffmpeg/libswscale/swscale.h"
//包含yuvlib头文件
#include "include/libyuv/libyuv.h"

#define LOGI(FORMAT,...) __android_log_print(4,"cj5785",FORMAT,##__VA_ARGS__);
#define LOGE(FORMAT,...) __android_log_print(6,"cj5785",FORMAT,##__VA_ARGS__);

JNIEXPORT void JNICALL Java_com_cj5785_ffmpegnativeplayer_NativePlayer_render
  (JNIEnv *env, jobject jobj, jstring jstr_path, jobject obj_surface)
{
	LOGE("%s", "开始");
	const char *input_cstr = (*env)->GetStringUTFChars(env, jstr_path, NULL);

	//1.注册组件
	av_register_all();

	AVFormatContext *pFormatCtx = avformat_alloc_context();
	//2.打开视频文件
	if(avformat_open_input(&pFormatCtx, input_cstr, NULL, NULL) != 0)
	{
		LOGE("%s", "打开文件失败!");
		return;
	}

	//3.获取视频相关信息
	if(avformat_find_stream_info(pFormatCtx, NULL) < 0)
	{
		LOGE("%s", "获取视频信息失败!");
		return;
	}

	int i = 0;
	int video_stream_index = -1;
	for (i = 0; i < pFormatCtx->nb_streams; i++) {
		if(pFormatCtx->streams[i]->codec->codec_type == AVMEDIA_TYPE_VIDEO)
		{
			video_stream_index = i;
			break;
		}
	}

	if (video_stream_index == -1)
	{
		LOGE("%s","找不到视频流\n");
		return;
	}
	//4.获取解码器
	AVCodecContext *pCodecCtx = pFormatCtx->streams[video_stream_index]->codec;
	AVCodec *pCodec = avcodec_find_decoder(pCodecCtx->codec_id);
	if(pCodec == NULL)
	{
		LOGE("%s", "无法解码!");
		return;
	}

	//5.打开解码器
	if(avcodec_open2(pCodecCtx, pCodec, NULL) < 0)
	{
		LOGE("%s", "解码失败!");
		return;
	}

	//6.以帧为单位读取视频文件
	AVPacket *packet = (AVPacket *)av_malloc(sizeof(AVPacket));
	AVFrame *pFrame = av_frame_alloc();
	AVFrame *pRGBFrame = av_frame_alloc();

	//native绘制
	//窗体设置
	ANativeWindow *nativeWindow = ANativeWindow_fromSurface(env, obj_surface);
	//缓冲区设置
	ANativeWindow_Buffer outBuffer;
	int len, got_frame, frame_count = 0;
	while(av_read_frame(pFormatCtx, packet) >= 0)
	{
		if(packet->stream_index == video_stream_index)
		{
			len = avcodec_decode_video2(pCodecCtx, pFrame, &got_frame, packet);
			if(len < 0)
			{
				LOGE("%s","解码错误!");
				return;
			}
			if(got_frame)
			{
				LOGI("解码第%d帧", frame_count++);
				//a.lock
				//设置缓冲区属性（宽，高，像素格式）
				ANativeWindow_setBuffersGeometry(nativeWindow, pCodecCtx->width, pCodecCtx->height, WINDOW_FORMAT_RGBA_8888);
				ANativeWindow_lock(nativeWindow, &outBuffer, NULL);
				//b.fix buffer
				//设置RGB的缓冲区以及属性（像素格式，宽高），RGB缓冲区和outBuffer.bits是同一块内存
				avpicture_fill((AVPicture *)pRGBFrame, outBuffer.bits, AV_PIX_FMT_RGBA, pCodecCtx->width, pCodecCtx->height);
				//YUV转化为RGB
				I420ToARGB(pFrame->data[0], pFrame->linesize[0],
						pFrame->data[2], pFrame->linesize[2],
						pFrame->data[1], pFrame->linesize[1],
						pRGBFrame->data[0], pRGBFrame->linesize[0],
						pCodecCtx->width, pCodecCtx->height);
				//c.unlock
				ANativeWindow_unlockAndPost(nativeWindow);
				usleep(16 * 1000);
			}
		}

		av_free_packet(packet);
	}
	ANativeWindow_release(nativeWindow);
	av_frame_free(&pFrame);
	av_frame_free(&pRGBFrame);
	avcodec_close(pCodecCtx);
	avformat_free_context(pFormatCtx);
	(*env)->ReleaseStringUTFChars(env, jstr_path, input_cstr);
}

调用native，使其能够播放

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
package com.cj5785.ffmpegnativeplayer;

import java.io.File;

import com.cj5785.ffmpegnativeplayer.view.MySurfaceView;

import android.app.Activity;
import android.os.Bundle;
import android.os.Environment;
import android.view.Surface;
import android.view.View;

public class MainActivity extends Activity {

	private NativePlayer player;
	private MySurfaceView mySurfaceView;
	
	@Override
	protected void onCreate(Bundle savedInstanceState) {
		super.onCreate(savedInstanceState);
		setContentView(R.layout.activity_main);
		mySurfaceView = (MySurfaceView) findViewById(R.id.surface_view);
		player = new NativePlayer();
	}

	public void mPlay(View view) {
		String input = Environment.getExternalStorageDirectory().getAbsolutePath() + File.separatorChar + "oneplus.mp4";
		Surface surface = mySurfaceView.getHolder().getSurface();
		player.render(input, surface);
	}
}

至此，已经可以编译生成apk了，在手机上测试也没有问题

更改布局和主活动，使其可以播放多个测试视频

MainActivity.java

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
package com.cj5785.ffmpegnativeplayer;

import java.io.File;

import com.cj5785.ffmpegnativeplayer.view.MySurfaceView;

import android.app.Activity;
import android.os.Bundle;
import android.os.Environment;
import android.view.Surface;
import android.view.View;
import android.widget.ArrayAdapter;
import android.widget.Spinner;

public class MainActivity extends Activity {

	private NativePlayer player;
	private MySurfaceView mySurfaceView;
	private Spinner sp_video;
	
	@Override
	protected void onCreate(Bundle savedInstanceState) {
		super.onCreate(savedInstanceState);
		setContentView(R.layout.activity_main);
		mySurfaceView = (MySurfaceView) findViewById(R.id.surface_view);
		sp_video = (Spinner)findViewById(R.id.sp_video);
		player = new NativePlayer();
		//视频列表
		String[] videoArray = getResources().getStringArray(R.array.video_list);
		ArrayAdapter<String> adapter = new ArrayAdapter<String>(this, 
				android.R.layout.simple_list_item_1, android.R.id.text1,videoArray);
		sp_video.setAdapter(adapter);
		
	}

	public void mPlay(View view) {
		String filename = sp_video.getSelectedItem().toString();
		String input = Environment.getExternalStorageDirectory().getAbsolutePath() + File.separatorChar + filename;
		Surface surface = mySurfaceView.getHolder().getSurface();
		player.render(input, surface);
	}
}

activity_main.xml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent" >

    <com.cj5785.ffmpegnativeplayer.view.MySurfaceView
        android:id="@+id/surface_view"
        android:layout_width="fill_parent"
        android:layout_height="fill_parent"/>
    <LinearLayout
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:orientation="horizontal">
        
        <Spinner 
            android:id="@+id/sp_video"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"/>
        
        <Button
        	android:layout_width="wrap_content"
        	android:layout_height="wrap_content"
        	android:text="开始"
        	android:onClick="mPlay" />
        
    </LinearLayout>

</FrameLayout>

在string.xml中添加数组值

1
2
3
4
5
6
<string-array name="video_list">
    <item>naxienian.mp4</item>
    <item>cuc_ieschool.mkv</item>
    <item>sintel.wmv</item>
    <item>Nocturne.m4a</item>
</string-array>

需要注意的问题

在native的实现过程中，I420ToARGB()方法在调用的时候，UV的位置是颠倒的，需要对调UV的位置
在这个示例程序中，旨在说明native是怎么绘制的，其代码存在严重不足，比如在主线程中绘制界面
部分视频会出现花屏现象，这个问题在后面多线程解码的时候会解决

Donate comment here